Skip to content

Instantly share code, notes, and snippets.

@fourbytes
Last active June 8, 2020 07:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save fourbytes/afe757a0aa3af61a82d6b706268fd5a5 to your computer and use it in GitHub Desktop.
Save fourbytes/afe757a0aa3af61a82d6b706268fd5a5 to your computer and use it in GitHub Desktop.
[2020-06-08 05:43:58.680537] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 7.5 (args: /usr/sbin/glusterfsd -s redrock --volfile-id data0.redrock.bricks-bricksredrock -p /var/run/gluster/vols/data0/redrock-bricks-bricksredrock.pid -S /var/run/gluster/ed66b80c50cd37cd.socket --brick-name /bricks/bricksredrock -l /var/log/glusterfs/bricks/bricks-bricksredrock.log --xlator-option *-posix.glusterd-uuid=5b64dea3-d862-4599-a1ad-e57a3a0d3144 --process-name brick --brick-port 49152 --xlator-option data0-server.listen-port=49152 --xlator-option transport.address-family=inet6)
[2020-06-08 05:43:58.681028] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 16008
[2020-06-08 05:43:58.684123] I [socket.c:958:__socket_server_bind] 0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9
[2020-06-08 05:43:58.689603] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-06-08 05:43:58.689700] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-06-08 05:43:58.695051] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2020-06-08 05:43:58.695148] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2020-06-08 05:43:58.695546] I [rpcsvc.c:2690:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2020-06-08 05:43:58.697120] I [socket.c:958:__socket_server_bind] 0-socket.data0-changelog: closing (AF_UNIX) reuse check socket 14
[2020-06-08 05:43:58.697298] I [trash.c:2450:init] 0-data0-trash: no option specified for 'eliminate', using NULL
Final graph:
+------------------------------------------------------------------------------+
1: volume data0-posix
2: type storage/posix
3: option glusterd-uuid 5b64dea3-d862-4599-a1ad-e57a3a0d3144
4: option directory /bricks/bricksredrock
5: option volume-id 4f249232-c457-495c-b241-82a0d6dce673
6: option fips-mode-rchecksum on
7: option shared-brick-count 1
8: option brick-uid 36
9: option brick-gid 36
10: option reserve-size 0
11: end-volume
12:
13: volume data0-trash
14: type features/trash
15: option trash-dir .trashcan
16: option brick-path /bricks/bricksredrock
17: option trash-internal-op off
18: subvolumes data0-posix
19: end-volume
20:
21: volume data0-changelog
22: type features/changelog
23: option changelog-brick /bricks/bricksredrock
24: option changelog-dir /bricks/bricksredrock/.glusterfs/changelogs
25: option changelog-barrier-timeout 120
26: subvolumes data0-trash
27: end-volume
28:
29: volume data0-bitrot-stub
30: type features/bitrot-stub
31: option export /bricks/bricksredrock
32: option bitrot disable
33: subvolumes data0-changelog
34: end-volume
35:
36: volume data0-access-control
37: type features/access-control
38: subvolumes data0-bitrot-stub
39: end-volume
40:
41: volume data0-locks
42: type features/locks
43: option enforce-mandatory-lock off
44: subvolumes data0-access-control
45: end-volume
46:
47: volume data0-worm
48: type features/worm
49: option worm off
50: option worm-file-level off
51: option worm-files-deletable on
52: subvolumes data0-locks
53: end-volume
54:
55: volume data0-read-only
56: type features/read-only
57: option read-only off
58: subvolumes data0-worm
59: end-volume
60:
61: volume data0-leases
62: type features/leases
63: option leases off
64: subvolumes data0-read-only
65: end-volume
66:
67: volume data0-upcall
68: type features/upcall
69: option cache-invalidation off
70: subvolumes data0-leases
71: end-volume
72:
73: volume data0-io-threads
74: type performance/io-threads
75: option low-prio-threads 32
76: subvolumes data0-upcall
77: end-volume
78:
79: volume data0-selinux
80: type features/selinux
81: option selinux on
82: subvolumes data0-io-threads
83: end-volume
84:
85: volume data0-marker
86: type features/marker
87: option volume-uuid 4f249232-c457-495c-b241-82a0d6dce673
88: option timestamp-file /var/lib/glusterd/vols/data0/marker.tstamp
89: option quota-version 0
90: option xtime off
91: option gsync-force-xtime off
92: option quota off
93: option inode-quota off
94: subvolumes data0-selinux
95: end-volume
96:
97: volume data0-barrier
98: type features/barrier
99: option barrier disable
100: option barrier-timeout 120
101: subvolumes data0-marker
102: end-volume
103:
104: volume data0-index
105: type features/index
106: option index-base /bricks/bricksredrock/.glusterfs/indices
107: option xattrop-dirty-watchlist trusted.afr.dirty
108: option xattrop-pending-watchlist trusted.afr.data0-
109: subvolumes data0-barrier
110: end-volume
111:
112: volume data0-quota
113: type features/quota
114: option volume-uuid data0
115: option server-quota off
116: option deem-statfs off
117: subvolumes data0-index
118: end-volume
119:
120: volume /bricks/bricksredrock
121: type debug/io-stats
122: option auth.addr./bricks/bricksredrock.allow *
123: option auth-path /bricks/bricksredrock
124: option auth.login.8c176837-1d40-4f59-a058-e5fbae607b15.password fba7f5d7-0083-4b4a-a023-8d0328a7a268
125: option auth.login./bricks/bricksredrock.allow 8c176837-1d40-4f59-a058-e5fbae607b15
126: option unique-id /bricks/bricksredrock
127: option log-level INFO
128: option threads 16
129: option latency-measurement off
130: option count-fop-hits off
131: option global-threading off
132: subvolumes data0-quota
133: end-volume
134:
135: volume data0-server
136: type protocol/server
137: option transport.socket.listen-port 49152
138: option rpc-auth.auth-glusterfs on
139: option rpc-auth.auth-unix on
140: option rpc-auth.auth-null on
141: option rpc-auth-allow-insecure on
142: option transport-type tcp
143: option transport.address-family inet
144: option auth.login./bricks/bricksredrock.allow 8c176837-1d40-4f59-a058-e5fbae607b15
145: option auth.login.8c176837-1d40-4f59-a058-e5fbae607b15.password fba7f5d7-0083-4b4a-a023-8d0328a7a268
146: option auth-path /bricks/bricksredrock
147: option auth.addr./bricks/bricksredrock.allow *
148: option transport.socket.keepalive 1
149: option transport.socket.ssl-enabled off
150: option event-threads 4
151: option transport.socket.keepalive-time 20
152: option transport.socket.keepalive-interval 2
153: option transport.socket.keepalive-count 9
154: option transport.listen-backlog 1024
155: subvolumes /bricks/bricksredrock
156: end-volume
157:
+------------------------------------------------------------------------------+
[2020-06-08 05:48:22.515848] W [socket.c:775:__socket_rwv] 0-glusterfs: readv on 2407:a080:7000::1:24007 failed (No data available)
[2020-06-08 05:48:22.515888] I [glusterfsd-mgmt.c:2719:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: redrock
[2020-06-08 05:48:25.519160] I [glusterfsd-mgmt.c:2282:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2020-06-08 05:49:23.287185] I [glusterfsd-mgmt.c:2739:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2020-06-08 05:49:39.051597] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x82de) [0x7f352e6d62de] -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xfd) [0x5d9df7932c1d] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x58) [0x5d9df7932a68] ) 0-: received signum (15), shutting down
[2020-06-08 05:51:28.344653] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 7.5 (args: /usr/sbin/glusterfsd -s redrock --volfile-id data0.redrock.bricks-bricksredrock -p /var/run/gluster/vols/data0/redrock-bricks-bricksredrock.pid -S /var/run/gluster/ed66b80c50cd37cd.socket --brick-name /bricks/bricksredrock -l /var/log/glusterfs/bricks/bricks-bricksredrock.log --xlator-option *-posix.glusterd-uuid=5b64dea3-d862-4599-a1ad-e57a3a0d3144 --process-name brick --brick-port 49152 --xlator-option data0-server.listen-port=49152 --xlator-option transport.address-family=inet6)
[2020-06-08 05:51:28.345328] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 4579
[2020-06-08 05:51:28.348255] I [socket.c:958:__socket_server_bind] 0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9
[2020-06-08 05:51:28.353614] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-06-08 05:51:28.353627] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-06-08 05:51:28.398920] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2020-06-08 05:51:28.399409] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2020-06-08 05:51:28.400497] I [rpcsvc.c:2690:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2020-06-08 05:51:28.408004] I [socket.c:958:__socket_server_bind] 0-socket.data0-changelog: closing (AF_UNIX) reuse check socket 14
[2020-06-08 05:51:28.408208] I [trash.c:2450:init] 0-data0-trash: no option specified for 'eliminate', using NULL
Final graph:
+------------------------------------------------------------------------------+
1: volume data0-posix
2: type storage/posix
3: option glusterd-uuid 5b64dea3-d862-4599-a1ad-e57a3a0d3144
4: option directory /bricks/bricksredrock
5: option volume-id 4f249232-c457-495c-b241-82a0d6dce673
6: option fips-mode-rchecksum on
7: option shared-brick-count 1
8: option brick-uid 36
9: option brick-gid 36
10: option reserve-size 0
11: end-volume
12:
13: volume data0-trash
14: type features/trash
15: option trash-dir .trashcan
16: option brick-path /bricks/bricksredrock
17: option trash-internal-op off
18: subvolumes data0-posix
19: end-volume
20:
21: volume data0-changelog
22: type features/changelog
23: option changelog-brick /bricks/bricksredrock
24: option changelog-dir /bricks/bricksredrock/.glusterfs/changelogs
25: option changelog-barrier-timeout 120
26: subvolumes data0-trash
27: end-volume
28:
29: volume data0-bitrot-stub
30: type features/bitrot-stub
31: option export /bricks/bricksredrock
32: option bitrot disable
33: subvolumes data0-changelog
34: end-volume
35:
36: volume data0-access-control
37: type features/access-control
38: subvolumes data0-bitrot-stub
39: end-volume
40:
41: volume data0-locks
42: type features/locks
43: option enforce-mandatory-lock off
44: subvolumes data0-access-control
45: end-volume
46:
47: volume data0-worm
48: type features/worm
49: option worm off
50: option worm-file-level off
51: option worm-files-deletable on
52: subvolumes data0-locks
53: end-volume
54:
55: volume data0-read-only
56: type features/read-only
57: option read-only off
58: subvolumes data0-worm
59: end-volume
60:
61: volume data0-leases
62: type features/leases
63: option leases off
64: subvolumes data0-read-only
65: end-volume
66:
67: volume data0-upcall
68: type features/upcall
69: option cache-invalidation off
70: subvolumes data0-leases
71: end-volume
72:
73: volume data0-io-threads
74: type performance/io-threads
75: option low-prio-threads 32
76: subvolumes data0-upcall
77: end-volume
78:
79: volume data0-selinux
80: type features/selinux
81: option selinux on
82: subvolumes data0-io-threads
83: end-volume
84:
85: volume data0-marker
86: type features/marker
87: option volume-uuid 4f249232-c457-495c-b241-82a0d6dce673
88: option timestamp-file /var/lib/glusterd/vols/data0/marker.tstamp
89: option quota-version 0
90: option xtime off
91: option gsync-force-xtime off
92: option quota off
93: option inode-quota off
94: subvolumes data0-selinux
95: end-volume
96:
97: volume data0-barrier
98: type features/barrier
99: option barrier disable
100: option barrier-timeout 120
101: subvolumes data0-marker
102: end-volume
103:
104: volume data0-index
105: type features/index
106: option index-base /bricks/bricksredrock/.glusterfs/indices
107: option xattrop-dirty-watchlist trusted.afr.dirty
108: option xattrop-pending-watchlist trusted.afr.data0-
109: subvolumes data0-barrier
110: end-volume
111:
112: volume data0-quota
113: type features/quota
114: option volume-uuid data0
115: option server-quota off
116: option deem-statfs off
117: subvolumes data0-index
118: end-volume
119:
120: volume /bricks/bricksredrock
121: type debug/io-stats
122: option auth.addr./bricks/bricksredrock.allow *
123: option auth-path /bricks/bricksredrock
124: option auth.login.8c176837-1d40-4f59-a058-e5fbae607b15.password fba7f5d7-0083-4b4a-a023-8d0328a7a268
125: option auth.login./bricks/bricksredrock.allow 8c176837-1d40-4f59-a058-e5fbae607b15
126: option unique-id /bricks/bricksredrock
127: option log-level INFO
128: option threads 16
129: option latency-measurement off
130: option count-fop-hits off
131: option global-threading off
132: subvolumes data0-quota
133: end-volume
134:
135: volume data0-server
136: type protocol/server
137: option transport.socket.listen-port 49152
138: option rpc-auth.auth-glusterfs on
139: option rpc-auth.auth-unix on
140: option rpc-auth.auth-null on
141: option rpc-auth-allow-insecure on
142: option transport-type tcp
143: option transport.address-family inet
144: option auth.login./bricks/bricksredrock.allow 8c176837-1d40-4f59-a058-e5fbae607b15
145: option auth.login.8c176837-1d40-4f59-a058-e5fbae607b15.password fba7f5d7-0083-4b4a-a023-8d0328a7a268
146: option auth-path /bricks/bricksredrock
147: option auth.addr./bricks/bricksredrock.allow *
148: option transport.socket.keepalive 1
149: option transport.socket.ssl-enabled off
150: option event-threads 4
151: option transport.socket.keepalive-time 20
152: option transport.socket.keepalive-interval 2
153: option transport.socket.keepalive-count 9
154: option transport.listen-backlog 1024
155: subvolumes /bricks/bricksredrock
156: end-volume
157:
+------------------------------------------------------------------------------+
[2020-06-08 07:06:46.243076] W [socket.c:775:__socket_rwv] 0-glusterfs: readv on 2407:a080:7000::1:24007 failed (No data available)
[2020-06-08 07:06:46.243113] I [glusterfsd-mgmt.c:2719:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: redrock
[2020-06-08 07:06:49.246246] I [glusterfsd-mgmt.c:2282:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2020-06-08 06:47:18.448195] I [MSGID: 104045] [glfs-master.c:80:notify] 0-gfapi: New graph 72656472-6f63-6b2d-3233-37352d323032 (0) coming up
[2020-06-08 06:47:18.448230] I [MSGID: 114020] [client.c:2436:notify] 0-data0-client-0: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume data0-client-0
2: type protocol/client
3: option ping-timeout 30
4: option remote-host redrock
5: option remote-subvolume /bricks/bricksredrock
6: option transport-type socket
7: option transport.address-family inet
8: option username 8c176837-1d40-4f59-a058-e5fbae607b15
9: option password fba7f5d7-0083-4b4a-a023-8d0328a7a268
10: option transport.socket.ssl-enabled off
11: option filter-O_DIRECT off
12: option event-threads 4
13: option transport.tcp-user-timeout 0
14: option transport.socket.keepalive-time 20
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-count 9
17: option send-gids true
18: end-volume
19:
20: volume data0-dht
21: type cluster/distribute
22: option lock-migration off
23: option force-migration off
24: subvolumes data0-client-0
25: end-volume
26:
27: volume data0-shard
28: type features/shard
29: subvolumes data0-dht
30: end-volume
31:
32: volume data0-utime
33: type features/utime
34: option noatime on
35: subvolumes data0-shard
36: end-volume
37:
38: volume data0-write-behind
39: type performance/write-behind
40: option strict-O_DIRECT on
41: subvolumes data0-utime
42: end-volume
43:
44: volume data0-readdir-ahead
45: type performance/readdir-ahead
46: option parallel-readdir off
47: option rda-request-size 131072
48: option rda-cache-limit 10MB
49: subvolumes data0-write-behind
50: end-volume
51:
52: volume data0-open-behind
53: type performance/open-behind
54: subvolumes data0-readdir-ahead
55: end-volume
56:
57: volume data0-md-cache
58: type performance/md-cache
59: subvolumes data0-open-behind
60: end-volume
61:
62: volume data0-io-threads
63: type performance/io-threads
64: option low-prio-threads 32
65: subvolumes data0-md-cache
66: end-volume
67:
68: volume data0
69: type debug/io-stats
70: option log-level INFO
71: option threads 16
72: option latency-measurement off
73: option count-fop-hits off
74: option global-threading off
75: subvolumes data0-io-threads
76: end-volume
77:
78: volume meta-autoload
79: type meta
80: subvolumes data0
81: end-volume
82:
+------------------------------------------------------------------------------+
[2020-06-08 06:47:18.453055] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-data0-client-0: changing port to 49152 (from 0)
[2020-06-08 06:47:18.453087] I [socket.c:865:__socket_shutdown] 0-data0-client-0: intentional socket shutdown(10)
[2020-06-08 06:48:11.908044] I [MSGID: 104045] [glfs-master.c:80:notify] 0-gfapi: New graph 72656472-6f63-6b2d-3330-34352d323032 (0) coming up
[2020-06-08 06:48:11.908092] I [MSGID: 114020] [client.c:2436:notify] 0-data0-client-0: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume data0-client-0
2: type protocol/client
3: option ping-timeout 30
4: option remote-host redrock
5: option remote-subvolume /bricks/bricksredrock
6: option transport-type socket
7: option transport.address-family inet
8: option username 8c176837-1d40-4f59-a058-e5fbae607b15
9: option password fba7f5d7-0083-4b4a-a023-8d0328a7a268
10: option transport.socket.ssl-enabled off
11: option filter-O_DIRECT off
12: option event-threads 4
13: option transport.tcp-user-timeout 0
14: option transport.socket.keepalive-time 20
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-count 9
17: option send-gids true
18: end-volume
19:
20: volume data0-dht
21: type cluster/distribute
22: option lock-migration off
23: option force-migration off
24: subvolumes data0-client-0
25: end-volume
26:
27: volume data0-shard
28: type features/shard
29: subvolumes data0-dht
30: end-volume
31:
32: volume data0-utime
33: type features/utime
34: option noatime on
35: subvolumes data0-shard
36: end-volume
37:
38: volume data0-write-behind
39: type performance/write-behind
40: option strict-O_DIRECT on
41: subvolumes data0-utime
42: end-volume
[2020-06-08 06:48:11.912905] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-data0-client-0: changing port to 49152 (from 0)
43:
44: volume data0-readdir-ahead
45: type performance/readdir-ahead
46: option parallel-readdir off
47: option rda-request-size 131072
48: option rda-cache-limit 10MB
[2020-06-08 06:48:11.912968] I [socket.c:865:__socket_shutdown] 0-data0-client-0: intentional socket shutdown(10)
49: subvolumes data0-write-behind
50: end-volume
51:
52: volume data0-open-behind
53: type performance/open-behind
54: subvolumes data0-readdir-ahead
55: end-volume
56:
57: volume data0-md-cache
58: type performance/md-cache
59: subvolumes data0-open-behind
60: end-volume
61:
62: volume data0-io-threads
63: type performance/io-threads
64: option low-prio-threads 32
65: subvolumes data0-md-cache
66: end-volume
67:
68: volume data0
69: type debug/io-stats
70: option log-level INFO
71: option threads 16
72: option latency-measurement off
73: option count-fop-hits off
74: option global-threading off
75: subvolumes data0-io-threads
76: end-volume
77:
78: volume meta-autoload
79: type meta
80: subvolumes data0
81: end-volume
82:
+------------------------------------------------------------------------------+
gluster> volume status data0
Status of volume: data0
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick redrock:/bricks/bricksredrock 49152 0 Y 4579
Task Status of Volume data0
------------------------------------------------------------------------------
There are no active volume tasks
gluster> volume heal data0 enable
Enable heal on volume data0 has been unsuccessful:
operation failed
gluster> volume heal data0 info
data0: Not able to fetch volfile from glusterd
Volume heal failed.
gluster> peer status
Number of Peers: 0
gluster> volume info data0
Volume Name: data0
Type: Distribute
Volume ID: 4f249232-c457-495c-b241-82a0d6dce673
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: redrock:/bricks/bricksredrock
Options Reconfigured:
network.ping-timeout: 30
performance.strict-o-direct: on
storage.owner-gid: 36
storage.owner-uid: 36
performance.client-io-threads: on
server.event-threads: 4
client.event-threads: 4
user.cifs: off
features.shard: on
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
i_get_volume] 0-management: Received get vol req
[2020-06-08 05:15:00.216635] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 05:20:00.130798] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 05:20:00.218770] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 05:25:00.133427] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 05:25:00.219597] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 05:30:00.132200] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 05:30:00.220423] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 05:35:00.132271] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 05:35:00.220230] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 05:40:00.130355] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 05:40:00.220474] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 05:42:07.299476] I [MSGID: 106499] [glusterd-handler.c:4264:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data0
[2020-06-08 05:42:10.340725] I [MSGID: 106542] [glusterd-utils.c:8689:glusterd_brick_signal] 0-glusterd: sending signal 15 to brick with pid 16999
[2020-06-08 05:42:11.340890] I [socket.c:865:__socket_shutdown] 0-management: intentional socket shutdown(14)
[2020-06-08 05:42:11.340917] I [MSGID: 106143] [glusterd-pmap.c:389:pmap_registry_remove] 0-pmap: removing brick /bricks/bricksredrock on port 49152
[2020-06-08 05:42:11.382995] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-06-08 05:42:11.383017] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: nfs service is stopped
[2020-06-08 05:42:11.383032] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2020-06-08 05:42:11.383134] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-06-08 05:42:11.383146] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped
[2020-06-08 05:42:11.383199] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already stopped
[2020-06-08 05:42:11.383211] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: scrub service is stopped
[2020-06-08 05:42:56.076818] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume bricks.rdma
[2020-06-08 05:42:56.077277] E [MSGID: 106176] [glusterd-handshake.c:1066:__server_getspec] 0-management: Failed to mount the volume
[2020-06-08 05:43:03.139742] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0.rdma
[2020-06-08 05:43:18.885966] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0.tcp
[2020-06-08 05:43:58.674582] I [glusterd-utils.c:6582:glusterd_brick_start] 0-management: starting a fresh brick process for brick /bricks/bricksredrock
[2020-06-08 05:43:58.689884] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0.redrock.bricks-bricksredrock
[2020-06-08 05:43:58.699017] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind] 0-pmap: adding brick /bricks/bricksredrock on port 49152
[2020-06-08 05:43:58.699579] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2020-06-08 05:43:58.739622] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-06-08 05:43:58.739650] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: nfs service is stopped
[2020-06-08 05:43:58.739663] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2020-06-08 05:43:58.739793] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-06-08 05:43:58.739818] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped
[2020-06-08 05:43:58.739876] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already stopped
[2020-06-08 05:43:58.739892] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: scrub service is stopped
[2020-06-08 05:43:03.140237] E [MSGID: 106176] [glusterd-handshake.c:1066:__server_getspec] 0-management: Failed to mount the volume
[2020-06-08 05:44:03.458375] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0.tcp
[2020-06-08 05:45:00.132697] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 05:45:00.219771] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 05:46:37.853337] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0.tcp
[2020-06-08 05:48:22.514138] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x82de) [0x76276a15c2de] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd) [0x5e65f5c2bc1d] -->/usr/sbin/glusterd(cleanup_and_exit+0x58) [0x5e65f5c2ba68] ) 0-: received signum (15), shutting down
[2020-06-08 05:48:22.521945] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 7.5 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2020-06-08 05:48:22.522378] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 18070
[2020-06-08 05:48:22.524211] I [MSGID: 106478] [glusterd.c:1426:init] 0-management: Maximum allowed open file descriptors set to 65536
[2020-06-08 05:48:22.524240] I [MSGID: 106479] [glusterd.c:1482:init] 0-management: Using /var/lib/glusterd as working directory
[2020-06-08 05:48:22.524249] I [MSGID: 106479] [glusterd.c:1488:init] 0-management: Using /var/run/gluster as pid file working directory
[2020-06-08 05:48:22.526919] I [socket.c:1015:__socket_server_bind] 0-socket.management: process started listening on port (24007)
[2020-06-08 05:48:22.532327] I [rdma.c:4742:gf_rdma_listen] 0-rdma.management: process started listening on port (51293)
[2020-06-08 05:48:22.533355] I [socket.c:958:__socket_server_bind] 0-socket.management: closing (AF_UNIX) reuse check socket 14
[2020-06-08 05:48:22.533683] I [MSGID: 106059] [glusterd.c:1865:init] 0-management: max-port override: 60999
[2020-06-08 05:48:24.815040] I [MSGID: 106513] [glusterd-store.c:2257:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 70200
[2020-06-08 05:48:24.815280] W [MSGID: 106204] [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2020-06-08 05:48:24.815421] W [MSGID: 106204] [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2020-06-08 05:48:24.889758] I [MSGID: 106544] [glusterd.c:152:glusterd_uuid_init] 0-management: retrieved UUID: 5b64dea3-d862-4599-a1ad-e57a3a0d3144
[2020-06-08 05:48:24.900456] I [MSGID: 106194] [glusterd-store.c:4102:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
[2020-06-08 05:48:24.901125] I [glusterd-utils.c:6495:glusterd_brick_start] 0-management: discovered already-running brick /bricks/bricksredrock
Final graph:
+------------------------------------------------------------------------------+
[2020-06-08 05:48:24.901144] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind] 0-pmap: adding brick /bricks/bricksredrock on port 49152
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.listen-backlog 1024
8: option max-port 60999
9: option transport.address-family inet6
10: option event-threads 1
11: option ping-timeout 0
12: option transport.rdma.listen-port 24008
13: option transport.socket.listen-port 24007
14: option transport.socket.read-fail-log off
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-time 10
17: option transport-type socket,rdma
18: option working-directory /var/lib/glusterd
19: end-volume
20:
+------------------------------------------------------------------------------+
[2020-06-08 05:48:24.901203] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2020-06-08 05:48:24.901399] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-06-08 05:48:24.942432] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2020-06-08 05:48:24.942537] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-06-08 05:48:24.942555] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: nfs service is stopped
[2020-06-08 05:48:24.942568] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2020-06-08 05:48:24.942601] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2020-06-08 05:48:24.942731] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already stopped
[2020-06-08 05:48:24.942746] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: quotad service is stopped
[2020-06-08 05:48:24.942777] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2020-06-08 05:48:24.942918] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-06-08 05:48:24.942937] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped
[2020-06-08 05:48:24.942969] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2020-06-08 05:48:24.943090] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already stopped
[2020-06-08 05:48:24.943104] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: scrub service is stopped
[2020-06-08 05:48:24.943139] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2020-06-08 05:48:24.943247] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600
[2020-06-08 05:48:25.518450] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0.tcp
[2020-06-08 05:48:25.518598] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0.redrock.bricks-bricksredrock
[2020-06-08 05:48:25.519140] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind] 0-pmap: adding brick /bricks/bricksredrock on port 49152
[2020-06-08 05:49:23.284572] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x82de) [0x78559e4012de] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd) [0x5f0f62e80c1d] -->/usr/sbin/glusterd(cleanup_and_exit+0x58) [0x5f0f62e80a68] ) 0-: received signum (15), shutting down
[2020-06-08 05:51:25.639390] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 7.5 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2020-06-08 05:51:25.640813] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 2689
[2020-06-08 05:51:25.663101] I [MSGID: 106478] [glusterd.c:1426:init] 0-management: Maximum allowed open file descriptors set to 65536
[2020-06-08 05:51:25.663140] I [MSGID: 106479] [glusterd.c:1482:init] 0-management: Using /var/lib/glusterd as working directory
[2020-06-08 05:51:25.663150] I [MSGID: 106479] [glusterd.c:1488:init] 0-management: Using /var/run/gluster as pid file working directory
[2020-06-08 05:51:25.666478] I [socket.c:1015:__socket_server_bind] 0-socket.management: process started listening on port (24007)
[2020-06-08 05:51:25.689285] I [rdma.c:4742:gf_rdma_listen] 0-rdma.management: process started listening on port (51293)
[2020-06-08 05:51:25.692778] I [socket.c:958:__socket_server_bind] 0-socket.management: closing (AF_UNIX) reuse check socket 14
[2020-06-08 05:51:25.694201] I [MSGID: 106059] [glusterd.c:1865:init] 0-management: max-port override: 60999
[2020-06-08 05:51:28.245938] I [MSGID: 106513] [glusterd-store.c:2257:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 70200
[2020-06-08 05:51:28.246830] W [MSGID: 106204] [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2020-06-08 05:51:28.246987] W [MSGID: 106204] [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2020-06-08 05:51:28.323640] I [MSGID: 106544] [glusterd.c:152:glusterd_uuid_init] 0-management: retrieved UUID: 5b64dea3-d862-4599-a1ad-e57a3a0d3144
[2020-06-08 05:51:28.336716] I [MSGID: 106194] [glusterd-store.c:4102:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.listen-backlog 1024
8: option max-port 60999
9: option transport.address-family inet6
10: option event-threads 1
11: option ping-timeout 0
12: option transport.rdma.listen-port 24008
13: option transport.socket.listen-port 24007
14: option transport.socket.read-fail-log off
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-time 10
17: option transport-type socket,rdma
18: option working-directory /var/lib/glusterd
19: end-volume
20:
+------------------------------------------------------------------------------+
[2020-06-08 05:51:28.337680] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-06-08 05:51:28.337808] I [glusterd-utils.c:6582:glusterd_brick_start] 0-management: starting a fresh brick process for brick /bricks/bricksredrock
[2020-06-08 05:51:28.341099] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2020-06-08 05:51:28.377244] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2020-06-08 05:51:28.377379] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-06-08 05:51:28.377407] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: nfs service is stopped
[2020-06-08 05:51:28.377431] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2020-06-08 05:51:28.377466] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2020-06-08 05:51:28.377606] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already stopped
[2020-06-08 05:51:28.377621] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: quotad service is stopped
[2020-06-08 05:51:28.377653] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2020-06-08 05:51:28.377779] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-06-08 05:51:28.377794] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped
[2020-06-08 05:51:28.377824] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2020-06-08 05:51:28.377957] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already stopped
[2020-06-08 05:51:28.377976] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: scrub service is stopped
[2020-06-08 05:51:28.378016] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2020-06-08 05:51:28.378151] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600
[2020-06-08 05:51:28.379817] I [MSGID: 106495] [glusterd-handler.c:2978:__glusterd_handle_getwd] 0-glusterd: Received getwd req
[2020-06-08 05:51:28.379959] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0.redrock.bricks-bricksredrock
[2020-06-08 05:51:28.412239] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind] 0-pmap: adding brick /bricks/bricksredrock on port 49152
[2020-06-08 05:51:50.108349] I [MSGID: 106487] [glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2020-06-08 05:52:38.580966] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0.tcp
[2020-06-08 05:54:57.419576] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0.tcp
[2020-06-08 05:55:00.229668] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 05:55:00.319190] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:00:00.155371] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:00:00.246217] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:05:00.136830] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:05:00.223856] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:10:00.145939] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:10:00.235434] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:15:00.150488] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:15:00.237027] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:20:00.142701] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:20:00.229225] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:25:00.143153] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:25:00.230699] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:30:00.141488] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:30:00.234427] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:35:00.148531] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:35:00.243087] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:40:00.141584] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:40:00.230382] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:45:00.142293] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:45:00.229673] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:46:44.163477] I [MSGID: 106499] [glusterd-handler.c:4264:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data0
[2020-06-08 06:47:12.054424] I [MSGID: 106533] [glusterd-volume-ops.c:934:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume data0
[2020-06-08 06:47:12.054461] E [MSGID: 106265] [glusterd-volume-ops.c:973:__glusterd_handle_cli_heal_volume] 0-management: operation failed
[2020-06-08 06:47:18.444736] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0
[2020-06-08 06:47:32.845584] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/7.5/rpc-transport/socket.so: undefined symbol: xlator_api
[2020-06-08 06:47:32.847110] W [MSGID: 101095] [xlator.c:210:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/7.5/xlator/nfs/server.so: cannot open shared object file: No such file or directory
[2020-06-08 06:47:32.855539] W [MSGID: 101095] [xlator.c:210:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/7.5/xlator/storage/bd.so: cannot open shared object file: No such file or directory
[2020-06-08 06:48:11.904638] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume data0
The message "E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/7.5/rpc-transport/socket.so: undefined symbol: xlator_api" repeated 7 times between [2020-06-08 06:47:32.845584] and [2020-06-08 06:47:32.845651]
The message "W [MSGID: 101095] [xlator.c:210:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/7.5/xlator/nfs/server.so: cannot open shared object file: No such file or directory" repeated 30 times between [2020-06-08 06:47:32.847110] and [2020-06-08 06:47:32.847334]
[2020-06-08 06:50:00.139474] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:50:00.234734] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:54:11.464850] I [MSGID: 106499] [glusterd-handler.c:4264:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data0
[2020-06-08 06:55:00.137394] I [MSGID: 106488] [glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-06-08 06:55:00.226595] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec] 0-management: Received mount request for volume /data0
[2020-06-08 06:54:23.303243] I [MSGID: 106499] [glusterd-handler.c:4264:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data0
./glusterd.log:[2020-06-07 17:35:42.750506] E [MSGID: 106218] [glusterd-rebalance.c:461:glusterd_rebalance_cmd_validate] 0-glusterd: Volume data0 is not a distribute type or contains only 1 brick
./glusterd.log:[2020-06-07 17:35:42.750537] E [MSGID: 106121] [glusterd-mgmt.c:967:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Rebalance on local node
./glusterd.log:[2020-06-07 17:35:42.750545] E [MSGID: 106121] [glusterd-mgmt.c:2161:glusterd_mgmt_v3_initiate_all_phases_with_brickop_phase] 0-management: Pre Validation Failed
./glusterd.log:[2020-06-07 17:43:26.374660] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/7.5/rpc-transport/socket.so: undefined symbol: xlator_api
./glusterd.log:[2020-06-07 17:53:10.509977] E [MSGID: 106121] [glusterd-mgmt.c:967:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Reset Brick on local node
./glusterd.log:[2020-06-07 17:53:10.509985] E [MSGID: 106121] [glusterd-replace-brick.c:656:glusterd_mgmt_v3_initiate_replace_brick_cmd_phases] 0-management: Pre Validation Failed
./glusterd.log:[2020-06-07 17:54:00.467407] E [MSGID: 106537] [glusterd-volume-ops.c:1578:glusterd_op_stage_start_volume] 0-management: Volume data0 already started
./glusterd.log:[2020-06-07 17:54:00.467444] E [MSGID: 106121] [glusterd-mgmt.c:967:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Start on local node
./glusterd.log:[2020-06-07 17:54:00.467453] E [MSGID: 106121] [glusterd-mgmt.c:2327:glusterd_mgmt_v3_initiate_all_phases] 0-management: Pre Validation Failed
./glusterd.log:[2020-06-07 17:54:07.254166] E [MSGID: 106028] [glusterd-utils.c:8656:glusterd_brick_signal] 0-glusterd: Unable to open pidfile: /var/run/gluster/vols/data0/redrock-bricks-bricksredrock.pid [No such file or directory]
./glusterd.log:[2020-06-07 17:56:14.514826] E [MSGID: 106265] [glusterd-volume-ops.c:973:__glusterd_handle_cli_heal_volume] 0-management: operation failed
./glusterd.log:[2020-06-07 17:56:37.127752] E [MSGID: 106301] [glusterd-syncop.c:1318:gd_stage_op_phase] 0-management: Staging of operation 'Volume Heal' failed on localhost : Self-heal-daemon is disabled. Heal will not be triggered on volume data0
./glusterd.log:[2020-06-07 17:56:46.921701] E [MSGID: 106265] [glusterd-volume-ops.c:973:__glusterd_handle_cli_heal_volume] 0-management: operation failed
./glusterd.log:[2020-06-08 05:42:56.077277] E [MSGID: 106176] [glusterd-handshake.c:1066:__server_getspec] 0-management: Failed to mount the volume
./glusterd.log:[2020-06-08 05:43:03.140237] E [MSGID: 106176] [glusterd-handshake.c:1066:__server_getspec] 0-management: Failed to mount the volume
./glusterd.log:[2020-06-08 06:47:12.054461] E [MSGID: 106265] [glusterd-volume-ops.c:973:__glusterd_handle_cli_heal_volume] 0-management: operation failed
./glusterd.log:[2020-06-08 06:47:32.845584] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/7.5/rpc-transport/socket.so: undefined symbol: xlator_api
./cli.log:[2020-06-07 17:33:19.842344] E [cli-cmd-parser.c:2180:cli_cmd_volume_remove_brick_parse] 0-cli: Unable to parse remove-brick CLI
./cli.log:[2020-06-07 17:34:55.594194] E [cli-cmd-parser.c:2180:cli_cmd_volume_remove_brick_parse] 0-cli: Unable to parse remove-brick CLI
./cli.log:[2020-06-07 17:35:04.440467] E [cli-cmd-parser.c:2180:cli_cmd_volume_remove_brick_parse] 0-cli: Unable to parse remove-brick CLI
./cli.log:[2020-06-07 17:35:10.905674] E [cli-cmd-parser.c:2180:cli_cmd_volume_remove_brick_parse] 0-cli: Unable to parse remove-brick CLI
./cli.log:[2020-06-07 17:35:26.896225] E [cli-cmd-parser.c:2180:cli_cmd_volume_remove_brick_parse] 0-cli: Unable to parse remove-brick CLI
./cli.log:[2020-06-07 17:36:18.815857] E [cli-cmd-parser.c:2180:cli_cmd_volume_remove_brick_parse] 0-cli: Unable to parse remove-brick CLI
./cli.log:[2020-06-07 17:52:42.449105] E [cli-cmd-parser.c:2324:cli_cmd_volume_reset_brick_parse] 0-cli: Unable to parse reset-brick CLI
./cli.log:[2020-06-07 17:52:44.828287] E [cli-cmd-parser.c:2324:cli_cmd_volume_reset_brick_parse] 0-cli: Unable to parse reset-brick CLI
./cli.log:[2020-06-07 17:53:00.882456] E [cli-cmd-parser.c:2324:cli_cmd_volume_reset_brick_parse] 0-cli: Unable to parse reset-brick CLI
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 17:30:42.288736] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 17:37:11.407679] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 17:46:16.525474] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 17:52:08.781009] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 17:56:22.734029] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:02:10.894007] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:07:09.902362] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:12:08.910355] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:17:12.013005] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:22:11.020872] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:27:10.029521] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:32:09.038481] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:37:12.141356] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:42:11.150083] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:47:10.158048] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:52:09.165020] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 18:57:12.270443] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:02:11.278012] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:07:10.285936] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:12:09.293306] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:17:12.398006] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:22:11.405047] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:27:10.412954] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:32:09.422450] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:37:12.524844] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:42:11.533000] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:47:10.541069] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:52:09.550315] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 19:57:08.556919] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:02:11.662394] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:07:10.670210] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:12:09.676896] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:17:08.685892] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:22:11.788904] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:27:10.796990] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:32:09.805037] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:37:08.813828] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:42:11.916872] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:47:10.925077] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:52:09.934501] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 20:57:08.940999] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:02:12.046399] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:07:11.054026] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:12:10.061880] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:17:09.069407] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:22:12.173268] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:27:11.181286] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:32:10.188883] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:37:09.198351] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:42:12.301857] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:47:11.309849] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:52:10.317016] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 21:57:09.326140] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:02:12.429992] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:07:11.437232] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:12:10.444962] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:17:09.453379] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:22:12.557892] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:27:11.565899] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:32:10.574184] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:37:09.581016] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:42:08.590598] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:47:11.693302] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:52:10.702007] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 22:57:09.710043] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:02:08.718013] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:07:11.821831] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:12:10.829325] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:17:09.837343] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:22:08.845989] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:27:11.949952] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:32:10.957986] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:37:09.965012] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:42:08.972968] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:47:12.077305] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:52:11.085054] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-07 23:57:10.094043] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:02:09.101016] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:07:12.206441] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:12:11.213435] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:17:10.221480] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:22:09.229084] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:27:12.333036] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:32:11.342362] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:37:10.350048] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:42:09.357394] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:47:12.462037] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:52:11.469004] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 00:57:10.476973] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:02:09.485456] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:07:12.590303] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:12:11.597998] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:17:10.604988] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:22:09.613431] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:27:08.622010] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:32:11.726084] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:37:10.733017] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:42:09.741994] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:47:08.749970] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:52:11.853307] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 01:57:10.861030] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:02:09.870081] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:07:08.877971] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:12:11.981497] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:17:10.989397] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:22:09.997010] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:27:09.006063] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:32:12.108891] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:37:11.117034] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:42:10.125015] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:47:09.133435] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:52:12.238402] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 02:57:11.245046] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:02:10.253372] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:07:09.262252] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:12:12.365987] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:17:11.373093] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:22:10.382083] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:27:09.390027] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:32:12.493045] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:37:11.501399] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:42:10.510060] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:47:09.518025] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:52:12.621072] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 03:57:11.629250] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:02:10.637059] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:07:09.646433] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:12:08.654021] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:17:11.757953] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:22:10.766009] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:27:09.773999] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:32:08.782013] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:37:11.885989] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:42:10.894413] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:47:09.901251] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:52:08.910397] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 04:57:12.014390] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:02:11.022009] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:07:10.029387] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:12:09.038265] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:17:12.142425] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:22:11.150015] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:27:10.157006] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:32:09.165024] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:37:12.269448] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:42:11.276830] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:47:10.285985] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 05:57:12.410176] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:02:11.417166] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:07:10.425532] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:12:09.433540] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:17:12.538139] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:22:11.545014] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:27:10.553529] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:32:09.562532] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:37:08.569708] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:42:11.673485] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:47:10.681644] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:52:09.690545] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 06:57:08.698172] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 07:02:11.801090] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./rhev-data-center-mnt-glusterSD-redrock.syd.fourbs.net:_data0.log:[2020-06-08 07:07:10.809156] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./mnt.log:[2020-06-08 05:42:56.077358] E [glusterfsd-mgmt.c:2217:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server
./mnt.log:[2020-06-08 05:42:56.077375] E [glusterfsd-mgmt.c:2416:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:bricks.rdma)
./mnt.log:[2020-06-08 05:43:03.140304] E [glusterfsd-mgmt.c:2217:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server
./mnt.log:[2020-06-08 05:43:03.140321] E [glusterfsd-mgmt.c:2416:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:data0.rdma)
./mnt.log:[2020-06-08 05:43:18.893650] E [MSGID: 114058] [client-handshake.c:1455:client_query_portmap_cbk] 0-data0-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
./mnt.log:[2020-06-08 05:43:18.894769] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./mnt.log:[2020-06-08 05:46:12.941050] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./mnt.log:[2020-06-08 05:48:48.589970] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./mnt.log:[2020-06-08 05:49:23.131687] E [glusterfsd-mgmt.c:2416:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:/data0.tcp)
./mnt.log:[2020-06-08 05:54:49.050089] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./mnt.log:[2020-06-08 05:57:08.314589] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
./mnt.log:[2020-06-08 07:05:16.121736] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
$ rm /var/log/glusterfs/mnt.log
$ sudo mount -t glusterfs -o transport=tcp redrock.syd.fourbs.net:data0 /mnt/
Mount failed. Check the log file for more details.
$ cat /var/log/glusterfs/mnt.log
[2020-06-08 07:14:11.319822] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.5 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=redrock --volfile-server-transport=tcp --volfile-id=data0.tcp /mnt)
[2020-06-08 07:14:11.320769] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 14387
[2020-06-08 07:14:11.328259] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-06-08 07:14:11.328310] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-06-08 07:14:11.331675] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2020-06-08 07:14:11.332144] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2020-06-08 07:14:11.332374] I [MSGID: 114020] [client.c:2436:notify] 0-data0-client-0: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume data0-client-0
2: type protocol/client
3: option ping-timeout 30
4: option remote-host redrock
5: option remote-subvolume /bricks/bricksredrock
6: option transport-type socket
7: option transport.address-family inet
8: option username 8c176837-1d40-4f59-a058-e5fbae607b15
9: option password fba7f5d7-0083-4b4a-a023-8d0328a7a268
10: option transport.socket.ssl-enabled off
11: option filter-O_DIRECT off
12: option event-threads 4
13: option transport.tcp-user-timeout 0
14: option transport.socket.keepalive-time 20
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-count 9
17: option send-gids true
18: end-volume
19:
20: volume data0-dht
21: type cluster/distribute
22: option lock-migration off
23: option force-migration off
24: subvolumes data0-client-0
25: end-volume
26:
27: volume data0-shard
28: type features/shard
29: subvolumes data0-dht
30: end-volume
31:
32: volume data0-utime
33: type features/utime
34: option noatime on
35: subvolumes data0-shard
36: end-volume
37:
38: volume data0-write-behind
39: type performance/write-behind
40: option strict-O_DIRECT on
41: subvolumes data0-utime
42: end-volume
43:
44: volume data0-readdir-ahead
45: type performance/readdir-ahead
46: option parallel-readdir off
47: option rda-request-size 131072
48: option rda-cache-limit 10MB
49: subvolumes data0-write-behind
50: end-volume
51:
52: volume data0-open-behind
53: type performance/open-behind
54: subvolumes data0-readdir-ahead
55: end-volume
56:
57: volume data0-md-cache
58: type performance/md-cache
59: subvolumes data0-open-behind
60: end-volume
61:
62: volume data0-io-threads
63: type performance/io-threads
64: option low-prio-threads 32
65: subvolumes data0-md-cache
66: end-volume
67:
68: volume data0
69: type debug/io-stats
70: option log-level INFO
71: option threads 16
72: option latency-measurement off
73: option count-fop-hits off
74: option global-threading off
75: subvolumes data0-io-threads
76: end-volume
77:
78: volume meta-autoload
79: type meta
80: subvolumes data0
81: end-volume
82:
+------------------------------------------------------------------------------+
[2020-06-08 07:14:11.336487] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-data0-client-0: changing port to 49152 (from 0)
[2020-06-08 07:14:11.336518] I [socket.c:865:__socket_shutdown] 0-data0-client-0: intentional socket shutdown(13)
[2020-06-08 07:16:19.673921] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.27
[2020-06-08 07:16:19.673948] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0
[2020-06-08 07:16:19.674101] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2020-06-08 07:16:19.674216] W [fuse-bridge.c:1276:fuse_attr_cbk] 0-glusterfs-fuse: 5: LOOKUP() / => -1 (Transport endpoint is not connected)
[2020-06-08 07:16:19.674263] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2020-06-08 07:16:19.674275] W [fuse-bridge.c:3880:fuse_statfs_resume] 0-glusterfs-fuse: 3: STATFS (00000000-0000-0000-0000-000000000001) resolution fail
[2020-06-08 07:16:19.676736] W [fuse-bridge.c:1276:fuse_attr_cbk] 0-glusterfs-fuse: 7: LOOKUP() / => -1 (Transport endpoint is not connected)
[2020-06-08 07:16:19.692985] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /mnt
[2020-06-08 07:16:19.693247] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x82de) [0x7a096e14f2de] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x609a592dac1d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x58) [0x609a592daa68] ) 0-: received signum (15), shutting down
[2020-06-08 07:16:19.693269] I [fuse-bridge.c:6898:fini] 0-fuse: Unmounting '/mnt'.
[2020-06-08 07:16:19.693277] I [fuse-bridge.c:6903:fini] 0-fuse: Closing fuse connection to '/mnt'.
[2020-06-08 06:45:00.222731] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.5 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=redrock.syd.fourbs.net --volfile-server=redrock --volfile-id=/data0 /rhev/data-center/mnt/glusterSD/redrock.syd.fourbs.net:_data0)
[2020-06-08 06:45:00.223678] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 1230
[2020-06-08 06:45:00.229118] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-06-08 06:45:00.229535] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-06-08 06:45:00.233062] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2020-06-08 06:45:00.233112] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2020-06-08 06:45:00.233360] I [MSGID: 114020] [client.c:2436:notify] 0-data0-client-0: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume data0-client-0
2: type protocol/client
3: option ping-timeout 30
4: option remote-host redrock
5: option remote-subvolume /bricks/bricksredrock
6: option transport-type socket
7: option transport.address-family inet
8: option username 8c176837-1d40-4f59-a058-e5fbae607b15
9: option password fba7f5d7-0083-4b4a-a023-8d0328a7a268
10: option transport.socket.ssl-enabled off
11: option filter-O_DIRECT off
12: option event-threads 4
13: option transport.tcp-user-timeout 0
14: option transport.socket.keepalive-time 20
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-count 9
17: option send-gids true
18: end-volume
19:
20: volume data0-dht
21: type cluster/distribute
22: option lock-migration off
23: option force-migration off
24: subvolumes data0-client-0
25: end-volume
26:
27: volume data0-shard
28: type features/shard
29: subvolumes data0-dht
30: end-volume
31:
32: volume data0-utime
33: type features/utime
34: option noatime on
35: subvolumes data0-shard
36: end-volume
37:
38: volume data0-write-behind
39: type performance/write-behind
40: option strict-O_DIRECT on
41: subvolumes data0-utime
42: end-volume
43:
44: volume data0-readdir-ahead
45: type performance/readdir-ahead
46: option parallel-readdir off
47: option rda-request-size 131072
48: option rda-cache-limit 10MB
49: subvolumes data0-write-behind
50: end-volume
51:
52: volume data0-open-behind
53: type performance/open-behind
54: subvolumes data0-readdir-ahead
55: end-volume
56:
57: volume data0-md-cache
58: type performance/md-cache
59: subvolumes data0-open-behind
60: end-volume
61:
[2020-06-08 06:45:00.237488] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-data0-client-0: changing port to 49152 (from 0)
62: volume data0-io-threads
63: type performance/io-threads
64: option low-prio-threads 32
65: subvolumes data0-md-cache
66: end-volume
67:
68: volume data0
69: type debug/io-stats
70: option log-level INFO
71: option threads 16
72: option latency-measurement off
[2020-06-08 06:45:00.237530] I [socket.c:865:__socket_shutdown] 0-data0-client-0: intentional socket shutdown(13)
73: option count-fop-hits off
74: option global-threading off
75: subvolumes data0-io-threads
76: end-volume
77:
78: volume meta-autoload
79: type meta
80: subvolumes data0
81: end-volume
82:
+------------------------------------------------------------------------------+
[2020-06-08 06:47:10.681475] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.27
[2020-06-08 06:47:10.681504] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0
[2020-06-08 06:47:10.681644] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2020-06-08 06:47:10.681754] W [fuse-bridge.c:1276:fuse_attr_cbk] 0-glusterfs-fuse: 5: LOOKUP() / => -1 (Transport endpoint is not connected)
[2020-06-08 06:47:10.681802] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2020-06-08 06:47:10.681813] W [fuse-bridge.c:3880:fuse_statfs_resume] 0-glusterfs-fuse: 3: STATFS (00000000-0000-0000-0000-000000000001) resolution fail
[2020-06-08 06:47:10.684509] W [fuse-bridge.c:1276:fuse_attr_cbk] 0-glusterfs-fuse: 7: LOOKUP() / => -1 (Transport endpoint is not connected)
[2020-06-08 06:47:10.698985] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /rhev/data-center/mnt/glusterSD/redrock.syd.fourbs.net:_data0
[2020-06-08 06:47:10.699276] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x82de) [0x7303b6ef22de] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x5e98e7241c1d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x58) [0x5e98e7241a68] ) 0-: received signum (15), shutting down
[2020-06-08 06:47:10.699304] I [fuse-bridge.c:6898:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/redrock.syd.fourbs.net:_data0'.
[2020-06-08 06:47:10.699313] I [fuse-bridge.c:6903:fini] 0-fuse: Closing fuse connection to '/rhev/data-center/mnt/glusterSD/redrock.syd.fourbs.net:_data0'.
[2020-06-08 06:50:00.226223] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.5 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=redrock.syd.fourbs.net --volfile-server=redrock --volfile-id=/data0 /rhev/data-center/mnt/glusterSD/redrock.syd.fourbs.net:_data0)
[2020-06-08 06:50:00.227241] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 3955
[2020-06-08 06:50:00.233936] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-06-08 06:50:00.234553] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-06-08 06:50:00.238156] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2020-06-08 06:50:00.238216] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2020-06-08 06:50:00.238479] I [MSGID: 114020] [client.c:2436:notify] 0-data0-client-0: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume data0-client-0
2: type protocol/client
3: option ping-timeout 30
4: option remote-host redrock
5: option remote-subvolume /bricks/bricksredrock
6: option transport-type socket
7: option transport.address-family inet
8: option username 8c176837-1d40-4f59-a058-e5fbae607b15
9: option password fba7f5d7-0083-4b4a-a023-8d0328a7a268
10: option transport.socket.ssl-enabled off
11: option filter-O_DIRECT off
12: option event-threads 4
13: option transport.tcp-user-timeout 0
14: option transport.socket.keepalive-time 20
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-count 9
17: option send-gids true
18: end-volume
19:
20: volume data0-dht
21: type cluster/distribute
22: option lock-migration off
23: option force-migration off
24: subvolumes data0-client-0
25: end-volume
26:
27: volume data0-shard
28: type features/shard
29: subvolumes data0-dht
30: end-volume
31:
32: volume data0-utime
33: type features/utime
34: option noatime on
35: subvolumes data0-shard
36: end-volume
37:
38: volume data0-write-behind
39: type performance/write-behind
40: option strict-O_DIRECT on
41: subvolumes data0-utime
42: end-volume
43:
44: volume data0-readdir-ahead
45: type performance/readdir-ahead
46: option parallel-readdir off
47: option rda-request-size 131072
48: option rda-cache-limit 10MB
49: subvolumes data0-write-behind
50: end-volume
51:
52: volume data0-open-behind
53: type performance/open-behind
54: subvolumes data0-readdir-ahead
55: end-volume
56:
57: volume data0-md-cache
58: type performance/md-cache
59: subvolumes data0-open-behind
60: end-volume
61:
62: volume data0-io-threads
63: type performance/io-threads
64: option low-prio-threads 32
65: subvolumes data0-md-cache
66: end-volume
67:
68: volume data0
69: type debug/io-stats
70: option log-level INFO
71: option threads 16
72: option latency-measurement off
73: option count-fop-hits off
74: option global-threading off
75: subvolumes data0-io-threads
76: end-volume
77:
78: volume meta-autoload
79: type meta
80: subvolumes data0
81: end-volume
82:
+------------------------------------------------------------------------------+
[2020-06-08 06:50:00.242703] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-data0-client-0: changing port to 49152 (from 0)
[2020-06-08 06:50:00.242747] I [socket.c:865:__socket_shutdown] 0-data0-client-0: intentional socket shutdown(13)
[2020-06-08 06:52:09.690355] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.27
[2020-06-08 06:52:09.690382] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0
[2020-06-08 06:52:09.690545] E [fuse-bridge.c:5235:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2020-06-08 06:52:09.690650] W [fuse-bridge.c:1276:fuse_attr_cbk] 0-glusterfs-fuse: 5: LOOKUP() / => -1 (Transport endpoint is not connected)
[2020-06-08 06:52:09.690698] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2020-06-08 06:52:09.690709] W [fuse-bridge.c:3880:fuse_statfs_resume] 0-glusterfs-fuse: 3: STATFS (00000000-0000-0000-0000-000000000001) resolution fail
[2020-06-08 06:52:09.693282] W [fuse-bridge.c:1276:fuse_attr_cbk] 0-glusterfs-fuse: 7: LOOKUP() / => -1 (Transport endpoint is not connected)
[2020-06-08 06:52:09.699988] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /rhev/data-center/mnt/glusterSD/redrock.syd.fourbs.net:_data0
[2020-06-08 06:52:09.700249] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x82de) [0x7ef896f8c2de] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x5f3bed31cc1d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x58) [0x5f3bed31ca68] ) 0-: received signum (15), shutting down
[2020-06-08 06:52:09.700271] I [fuse-bridge.c:6898:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/redrock.syd.fourbs.net:_data0'.
[2020-06-08 06:52:09.700279] I [fuse-bridge.c:6903:fini] 0-fuse: Closing fuse connection to '/rhev/data-center/mnt/glusterSD/redrock.syd.fourbs.net:_data0'.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment