Created
January 22, 2018 19:22
-
-
Save nh2/4e04a9a5d168f28ee925074a6b9c1170 to your computer and use it in GitHub Desktop.
GlusterFS "starting fresh brick process" happening twice
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [2018-01-22 04:40:50.733723] I [glusterd-utils.c:5926:glusterd_brick_start] 0-management: starting a fresh brick process for brick /data/glusterfs/myvol/brick1/brick | |
| [2018-01-22 04:40:50.734917] D [MSGID: 0] [glusterd-utils.c:1093:glusterd_brickinfo_new] 0-management: Returning 0 | |
| [2018-01-22 04:40:50.734944] D [MSGID: 0] [glusterd-utils.c:1068:glusterd_brickprocess_new] 0-management: Returning 0 | |
| [2018-01-22 04:40:50.734967] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 | |
| [2018-01-22 04:40:50.734975] D [rpc-clnt.c:1061:rpc_clnt_connection_init] 0-management: disable ping-timeout | |
| [2018-01-22 04:40:50.734985] D [rpc-transport.c:279:rpc_transport_load] 0-rpc-transport: attempt to load file /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/glusterfs/3.12.3/rpc-transport/socket.so | |
| [2018-01-22 04:40:50.735063] D [socket.c:4164:socket_init] 0-management: disabling nodelay | |
| [2018-01-22 04:40:50.735075] D [socket.c:4203:socket_init] 0-management: Configued transport.tcp-user-timeout=0 | |
| [2018-01-22 04:40:50.735081] D [socket.c:4221:socket_init] 0-management: Reconfigued transport.keepalivecnt=9 | |
| [2018-01-22 04:40:50.735089] D [socket.c:4306:socket_init] 0-management: SSL support on the I/O path is NOT enabled | |
| [2018-01-22 04:40:50.735095] I [socket.c:4309:socket_init] 0-management: SSL support for glusterd is ENABLED | |
| [2018-01-22 04:40:50.735101] D [socket.c:4326:socket_init] 0-management: using system polling thread | |
| [2018-01-22 04:40:50.735583] D [MSGID: 0] [glusterd-handler.c:3336:glusterd_rpc_create] 0-management: returning 0 | |
| [2018-01-22 04:40:50.735599] D [MSGID: 0] [glusterd-utils.c:1907:glusterd_brick_connect] 0-glusterd: Returning 0 | |
| [2018-01-22 04:40:50.735610] D [MSGID: 0] [glusterd-utils.c:5943:glusterd_brick_start] 0-management: returning 0 | |
| [2018-01-22 04:40:50.735612] D [MSGID: 0] [common-utils.c:347:gf_resolve_ip6] 0-resolver: returning ip-10.0.0.3 (port-24007) for hostname: 10.0.0.3 and port: 24007 | |
| [2018-01-22 04:40:50.735627] D [socket.c:3052:socket_fix_ssl_opts] 0-management: enabling SSL for portmapper connection | |
| [2018-01-22 04:40:50.735629] E [MSGID: 106155] [glusterd-utils.c:367:glusterd_unlock] 0-management: Cluster lock not held! | |
| [2018-01-22 04:40:50.735888] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libglusterfs.so.0(_gf_log_callingfn+0x13e)[0x7f242a52fcae] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f242a2faca4] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xd4)[0x7f242a2f6d34] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f242a2f7818] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f242a2f3a83] ))))) 0-: : ping timer event already removed | |
| [2018-01-22 04:40:54.488403] D [MSGID: 0] [common-utils.c:347:gf_resolve_ip6] 0-resolver: returning ip-10.0.0.1 (port-24007) for hostname: 10.0.0.1 and port: 24007 | |
| [2018-01-22 04:40:54.488457] D [socket.c:3052:socket_fix_ssl_opts] 0-management: enabling SSL for portmapper connection | |
| [2018-01-22 04:40:54.489288] E [socket.c:3289:socket_connect] 0-management: connection attempt on failed, (Connection refused) | |
| [2018-01-22 04:40:54.489335] D [MSGID: 0] [socket.c:3358:socket_connect] 0-management: socket connect is failed so close it | |
| [2018-01-22 04:40:54.489512] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:40:54.489854] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libglusterfs.so.0(_gf_log_callingfn+0x13e)[0x7f242a52fcae] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f242a2faca4] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xd4)[0x7f242a2f6d34] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f242a2f7818] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f242a2f3a83] ))))) 0-: : ping timer event already removed | |
| [2018-01-22 04:40:57.494598] D [MSGID: 0] [common-utils.c:347:gf_resolve_ip6] 0-resolver: returning ip-10.0.0.1 (port-24007) for hostname: 10.0.0.1 and port: 24007 | |
| [2018-01-22 04:40:57.494662] D [socket.c:3052:socket_fix_ssl_opts] 0-management: enabling SSL for portmapper connection | |
| [2018-01-22 04:40:57.495414] E [socket.c:3289:socket_connect] 0-management: connection attempt on failed, (Connection refused) | |
| [2018-01-22 04:40:57.495451] D [MSGID: 0] [socket.c:3358:socket_connect] 0-management: socket connect is failed so close it | |
| [2018-01-22 04:40:57.495605] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:40:57.495977] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libglusterfs.so.0(_gf_log_callingfn+0x13e)[0x7f242a52fcae] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f242a2faca4] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xd4)[0x7f242a2f6d34] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f242a2f7818] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f242a2f3a83] ))))) 0-: : ping timer event already removed | |
| [2018-01-22 04:41:00.498194] D [MSGID: 0] [common-utils.c:347:gf_resolve_ip6] 0-resolver: returning ip-10.0.0.1 (port-24007) for hostname: 10.0.0.1 and port: 24007 | |
| [2018-01-22 04:41:00.498224] D [socket.c:3052:socket_fix_ssl_opts] 0-management: enabling SSL for portmapper connection | |
| [2018-01-22 04:41:00.498658] E [socket.c:3289:socket_connect] 0-management: connection attempt on failed, (Connection refused) | |
| [2018-01-22 04:41:00.498673] D [MSGID: 0] [socket.c:3358:socket_connect] 0-management: socket connect is failed so close it | |
| [2018-01-22 04:41:00.498768] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:00.498946] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libglusterfs.so.0(_gf_log_callingfn+0x13e)[0x7f242a52fcae] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f242a2faca4] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xd4)[0x7f242a2f6d34] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f242a2f7818] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f242a2f3a83] ))))) 0-: : ping timer event already removed | |
| [2018-01-22 04:41:03.500312] D [MSGID: 0] [common-utils.c:347:gf_resolve_ip6] 0-resolver: returning ip-10.0.0.1 (port-24007) for hostname: 10.0.0.1 and port: 24007 | |
| [2018-01-22 04:41:03.500339] D [socket.c:3052:socket_fix_ssl_opts] 0-management: enabling SSL for portmapper connection | |
| [2018-01-22 04:41:03.500667] E [socket.c:3289:socket_connect] 0-management: connection attempt on failed, (Connection refused) | |
| [2018-01-22 04:41:03.500680] D [MSGID: 0] [socket.c:3358:socket_connect] 0-management: socket connect is failed so close it | |
| [2018-01-22 04:41:03.500794] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:03.500911] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libglusterfs.so.0(_gf_log_callingfn+0x13e)[0x7f242a52fcae] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f242a2faca4] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xd4)[0x7f242a2f6d34] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f242a2f7818] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f242a2f3a83] ))))) 0-: : ping timer event already removed | |
| [2018-01-22 04:41:05.975989] D [MSGID: 0] [glusterd-handler.c:3336:glusterd_rpc_create] 0-management: returning 0 | |
| [2018-01-22 04:41:05.976057] D [MSGID: 0] [glusterd-store.c:4462:glusterd_store_retrieve_peers] 0-management: Returning with 0 | |
| [2018-01-22 04:41:05.984377] D [socket.c:348:ssl_setup_connection] 0-management: peer CN = gluster-server | |
| [2018-01-22 04:41:05.984391] D [socket.c:351:ssl_setup_connection] 0-management: SSL verification succeeded (client: ) (server: ) | |
| [2018-01-22 04:41:05.984408] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.984417] D [MSGID: 0] [glusterd-handler.c:6279:__glusterd_peer_rpc_notify] 0-management: got RPC_CLNT_CONNECT | |
| [2018-01-22 04:41:05.984460] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.984499] D [rpc-clnt-ping.c:317:rpc_clnt_start_ping] 0-management: ping timeout is 0, returning | |
| [2018-01-22 04:41:05.985281] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk | |
| [2018-01-22 04:41:05.985277] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.985281] D [MSGID: 0] [glusterd-handshake.c:2218:__glusterd_peer_dump_version_cbk] 0-management: Proceeding to op-version handshake with peer 10.0.0.3 | |
| [2018-01-22 04:41:05.985308] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.985341] D [rpc-clnt-ping.c:317:rpc_clnt_start_ping] 0-management: ping timeout is 0, returning | |
| [2018-01-22 04:41:05.986021] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk | |
| [2018-01-22 04:41:05.986015] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.986020] D [MSGID: 0] [glusterd-handshake.c:1782:gd_validate_peer_op_version] 0-management: Peer 10.0.0.3 accepted | |
| [2018-01-22 04:41:05.986051] D [rpc-clnt-ping.c:317:rpc_clnt_start_ping] 0-management: ping timeout is 0, returning | |
| [2018-01-22 04:41:05.988836] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.988852] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk | |
| [2018-01-22 04:41:05.988850] D [MSGID: 0] [glusterd-peer-utils.c:1023:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.988852] D [MSGID: 0] [glusterd-sm.c:1283:glusterd_friend_sm_inject_event] 0-glusterd: Enqueue event: 'GD_FRIEND_EVENT_CONNECTED' | |
| [2018-01-22 04:41:05.988866] D [MSGID: 0] [glusterd-handshake.c:1735:glusterd_event_connected_inject] 0-glusterd: returning 0 | |
| [2018-01-22 04:41:05.988872] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.988876] D [MSGID: 0] [glusterd-sm.c:1375:glusterd_friend_sm] 0-glusterd: Dequeued event of type: 'GD_FRIEND_EVENT_CONNECTED' | |
| [2018-01-22 04:41:05.988881] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.989017] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk | |
| [2018-01-22 04:41:05.988885] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.989017] D [MSGID: 0] [glusterd-utils.c:3157:glusterd_add_volume_to_dict] 0-management: Returning with 0 | |
| [2018-01-22 04:41:05.989144] D [rpc-clnt-ping.c:317:rpc_clnt_start_ping] 0-management: ping timeout is 0, returning | |
| [2018-01-22 04:41:05.989173] D [MSGID: 0] [glusterd-rpc-ops.c:1632:glusterd_rpc_friend_add] 0-glusterd: Returning 0 | |
| [2018-01-22 04:41:05.989178] D [MSGID: 0] [glusterd-sm.c:334:glusterd_ac_friend_add] 0-glusterd: Returning with 0 | |
| [2018-01-22 04:41:05.989183] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.989188] D [MSGID: 0] [glusterd-utils.c:7545:glusterd_sm_tr_log_transition_add] 0-management: Transitioning from 'Peer in Cluster' to 'Peer in Cluster' due to event 'GD_FRIEND_EVENT_CONNECTED' | |
| [2018-01-22 04:41:05.989192] D [MSGID: 0] [glusterd-utils.c:7547:glusterd_sm_tr_log_transition_add] 0-management: returning 0 | |
| [2018-01-22 04:41:05.989196] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.989208] D [MSGID: 0] [glusterd-store.c:4147:glusterd_store_create_peer_dir] 0-glusterd: Returning with 0 | |
| [2018-01-22 04:41:05.989256] D [MSGID: 0] [store.c:378:gf_store_save_value] 0-management: returning: 0 | |
| [2018-01-22 04:41:05.989269] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk | |
| The message "D [MSGID: 0] [store.c:378:gf_store_save_value] 0-management: returning: 0" repeated 2 times between [2018-01-22 04:41:05.989256] and [2018-01-22 04:41:05.989269] | |
| [2018-01-22 04:41:05.989269] D [MSGID: 0] [glusterd-store.c:4276:glusterd_store_peer_write] 0-glusterd: Returning with 0 | |
| [2018-01-22 04:41:05.993129] D [MSGID: 0] [glusterd-store.c:3845:glusterd_store_retrieve_missed_snaps_list] 0-management: EOF for missed_snap_list | |
| [2018-01-22 04:41:05.993143] D [MSGID: 0] [glusterd-store.c:3952:glusterd_store_retrieve_snaps] 0-management: Returning with 0 | |
| [2018-01-22 04:41:05.993150] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.993154] D [MSGID: 0] [glusterd-utils.c:1141:glusterd_resolve_brick] 0-management: Returning 0 | |
| [2018-01-22 04:41:05.993159] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk | |
| [2018-01-22 04:41:05.993158] D [MSGID: 0] [glusterd-utils.c:1141:glusterd_resolve_brick] 0-management: Returning 0 | |
| [2018-01-22 04:41:05.993159] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:05.993169] D [MSGID: 0] [glusterd-utils.c:1141:glusterd_resolve_brick] 0-management: Returning 0 | |
| [2018-01-22 04:41:05.993174] D [MSGID: 0] [glusterd-store.c:4691:glusterd_restore] 0-management: Returning 0 | |
| [2018-01-22 04:41:05.993222] D [MSGID: 0] [options.c:1257:xlator_option_init_int32] 0-management: option event-threads using set value 1 | |
| Final graph: | |
| +------------------------------------------------------------------------------+ | |
| 1: volume management | |
| 2: type mgmt/glusterd | |
| 3: option transport.socket.ssl-enabled on | |
| 4: option rpc-auth.auth-glusterfs on | |
| 5: option rpc-auth.auth-unix on | |
| 6: option rpc-auth.auth-null on | |
| 7: option rpc-auth-allow-insecure on | |
| 8: option transport.listen-backlog 10 | |
| 9: option event-threads 1 | |
| 10: option ping-timeout 0 | |
| 11: option transport.socket.read-fail-log off | |
| 12: option transport.socket.keepalive-interval 2 | |
| 13: option transport.socket.keepalive-time 10 | |
| 14: option transport-type rdma | |
| 15: option working-directory /var/lib/glusterd | |
| 16: end-volume | |
| 17: | |
| +------------------------------------------------------------------------------+ | |
| [2018-01-22 04:41:05.993327] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 | |
| [2018-01-22 04:41:06.000890] D [socket.c:348:ssl_setup_connection] 0-socket.management: peer CN = gluster-client | |
| [2018-01-22 04:41:06.000902] D [socket.c:351:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: 10.0.2.1:49150) (server: 10.0.0.2:24007) | |
| [2018-01-22 04:41:06.092980] D [MSGID: 0] [glusterd-store.c:4302:glusterd_store_perform_peer_store] 0-glusterd: Returning 0 | |
| [2018-01-22 04:41:06.093011] D [MSGID: 0] [glusterd-store.c:4323:glusterd_store_peerinfo] 0-glusterd: Returning with 0 | |
| [2018-01-22 04:41:06.093043] D [MSGID: 0] [glusterd-utils.c:1669:glusterd_volinfo_find_by_volume_id] 0-management: Volume myvol found | |
| [2018-01-22 04:41:06.093078] D [MSGID: 0] [glusterd-utils.c:1093:glusterd_brickinfo_new] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.093092] D [MSGID: 0] [glusterd-utils.c:1297:glusterd_brickinfo_new_from_brick] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.093322] D [MSGID: 0] [glusterd-peer-utils.c:132:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: 10.0.0.2 | |
| [2018-01-22 04:41:06.093344] D [MSGID: 0] [common-utils.c:3523:gf_is_local_addr] 0-management: 10.0.0.2 | |
| [2018-01-22 04:41:06.093548] D [MSGID: 0] [common-utils.c:3408:gf_interface_search] 0-management: 10.0.0.2 is local address at interface tinc.myvpn | |
| [2018-01-22 04:41:06.093561] D [MSGID: 0] [glusterd-peer-utils.c:167:glusterd_hostname_to_uuid] 0-management: returning 0 | |
| [2018-01-22 04:41:06.093570] D [MSGID: 0] [glusterd-utils.c:1593:glusterd_volume_brickinfo_get] 0-management: Found brick 10.0.0.2:/data/glusterfs/myvol/brick1/brick in volume myvol | |
| [2018-01-22 04:41:06.093577] D [MSGID: 0] [glusterd-utils.c:1602:glusterd_volume_brickinfo_get] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.093590] D [MSGID: 0] [store.c:463:gf_store_handle_destroy] 0-: Returning 0 | |
| [2018-01-22 04:41:06.093598] D [MSGID: 0] [glusterd-utils.c:1628:glusterd_volume_brickinfo_get_by_brick] 0-glusterd: Returning 0 | |
| [2018-01-22 04:41:06.093607] D [MSGID: 0] [glusterd-utils.c:1669:glusterd_volinfo_find_by_volume_id] 0-management: Volume myvol found | |
| [2018-01-22 04:41:06.093614] D [MSGID: 0] [glusterd-handler.c:6038:__glusterd_brick_rpc_notify] 0-management: Connected to 10.0.0.2:/data/glusterfs/myvol/brick1/brick | |
| [2018-01-22 04:41:06.093622] D [MSGID: 0] [glusterd-utils.c:6380:glusterd_set_brick_status] 0-glusterd: Setting brick 10.0.0.2:/data/glusterfs/myvol/brick1/brick status to started | |
| [2018-01-22 04:41:06.093668] I [socket.c:2475:socket_event_handler] 0-transport: EPOLLERR - disconnecting now | |
| [2018-01-22 04:41:06.093689] D [MSGID: 0] [glusterd-utils.c:1669:glusterd_volinfo_find_by_volume_id] 0-management: Volume myvol found | |
| [2018-01-22 04:41:06.093715] D [MSGID: 0] [glusterd-utils.c:1093:glusterd_brickinfo_new] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.093732] D [MSGID: 0] [glusterd-utils.c:1297:glusterd_brickinfo_new_from_brick] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.093752] D [MSGID: 0] [glusterd-peer-utils.c:132:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: 10.0.0.2 | |
| [2018-01-22 04:41:06.093767] D [MSGID: 0] [common-utils.c:3523:gf_is_local_addr] 0-management: 10.0.0.2 | |
| [2018-01-22 04:41:06.093977] D [MSGID: 0] [common-utils.c:3408:gf_interface_search] 0-management: 10.0.0.2 is local address at interface tinc.myvpn | |
| [2018-01-22 04:41:06.093991] D [MSGID: 0] [glusterd-peer-utils.c:167:glusterd_hostname_to_uuid] 0-management: returning 0 | |
| [2018-01-22 04:41:06.093999] D [MSGID: 0] [glusterd-utils.c:1593:glusterd_volume_brickinfo_get] 0-management: Found brick 10.0.0.2:/data/glusterfs/myvol/brick1/brick in volume myvol | |
| [2018-01-22 04:41:06.094016] D [MSGID: 0] [glusterd-utils.c:1602:glusterd_volume_brickinfo_get] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.094023] D [MSGID: 0] [store.c:463:gf_store_handle_destroy] 0-: Returning 0 | |
| [2018-01-22 04:41:06.094042] D [MSGID: 0] [glusterd-utils.c:1628:glusterd_volume_brickinfo_get_by_brick] 0-glusterd: Returning 0 | |
| [2018-01-22 04:41:06.094048] I [MSGID: 106005] [glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management: Brick 10.0.0.2:/data/glusterfs/myvol/brick1/brick has disconnected from glusterd. | |
| [2018-01-22 04:41:06.094055] D [MSGID: 0] [glusterd-utils.c:1669:glusterd_volinfo_find_by_volume_id] 0-management: Volume myvol found | |
| [2018-01-22 04:41:06.094103] E [MSGID: 101012] [common-utils.c:3861:gf_is_service_running] 0-: Unable to read pidfile: /var/run/gluster/vols/myvol/10.0.0.2-data-glusterfs-myvol-brick1-brick.pid | |
| [2018-01-22 04:41:06.094125] D [MSGID: 0] [glusterd-utils.c:6383:glusterd_set_brick_status] 0-glusterd: Setting brick 10.0.0.2:/data/glusterfs/myvol/brick1/brick status to stopped | |
| [2018-01-22 04:41:06.094271] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libglusterfs.so.0(_gf_log_callingfn+0x13e)[0x7f242a52fcae] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f242a2faca4] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xd4)[0x7f242a2f6d34] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7f242a2f7818] (--> /nix/store/ad3g93m8mi3sijk032zxxc2maprqkhs6-glusterfs-3.12.3/lib/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f242a2f3a83] ))))) 0-: /var/run/gluster/1d7dda2d28e9fd88ad87ad8fe3d03f79.socket: ping timer event already removed | |
| [2018-01-22 04:41:06.094456] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvol | |
| [2018-01-22 04:41:06.094505] D [MSGID: 0] [glusterd-op-sm.c:209:glusterd_generate_txn_id] 0-management: Transaction_id = d3ffae58-1e42-43c9-9bec-6e0b4029e195 | |
| [2018-01-22 04:41:06.094534] D [MSGID: 0] [glusterd-op-sm.c:306:glusterd_set_txn_opinfo] 0-management: Successfully set opinfo for transaction ID : d3ffae58-1e42-43c9-9bec-6e0b4029e195 | |
| [2018-01-22 04:41:06.094546] D [MSGID: 0] [glusterd-op-sm.c:313:glusterd_set_txn_opinfo] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.094560] D [MSGID: 0] [glusterd-syncop.c:1855:gd_sync_task_begin] 0-management: Transaction ID : d3ffae58-1e42-43c9-9bec-6e0b4029e195 | |
| [2018-01-22 04:41:06.094584] D [MSGID: 0] [glusterd-locks.c:607:glusterd_mgmt_v3_lock] 0-management: Trying to acquire lock of vol myvol for 980f9026-d35c-44e3-b432-5b0beee53635 as myvol_vol | |
| [2018-01-22 04:41:06.094744] D [MSGID: 0] [glusterd-locks.c:698:glusterd_mgmt_v3_lock] 0-management: Lock for vol myvol successfully held by 980f9026-d35c-44e3-b432-5b0beee53635 | |
| [2018-01-22 04:41:06.094794] D [rpc-clnt-ping.c:317:rpc_clnt_start_ping] 0-management: ping timeout is 0, returning | |
| [2018-01-22 04:41:06.094803] D [MSGID: 0] [glusterd-syncop.c:431:gd_syncop_mgmt_v3_lock] 0-glusterd: Returning 0 | |
| [2018-01-22 04:41:06.095124] E [socket.c:2641:socket_poller] 0-socket.management: socket_poller 10.0.2.1:49150 failed (Input/output error) | |
| [2018-01-22 04:41:06.098145] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 0406f2c7-9146-429e-850d-f4637363ba51, host: 10.0.0.3, port: 0 | |
| [2018-01-22 04:41:06.098167] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:06.098174] D [MSGID: 0] [glusterd-sm.c:1283:glusterd_friend_sm_inject_event] 0-glusterd: Enqueue event: 'GD_FRIEND_EVENT_RCVD_ACC' | |
| [2018-01-22 04:41:06.098178] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:06.098182] D [MSGID: 0] [glusterd-sm.c:1375:glusterd_friend_sm] 0-glusterd: Dequeued event of type: 'GD_FRIEND_EVENT_RCVD_ACC' | |
| [2018-01-22 04:41:06.098187] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:06.098205] D [MSGID: 0] [glusterd-peer-utils.c:643:gd_add_friend_to_dict] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.098242] D [rpc-clnt-ping.c:317:rpc_clnt_start_ping] 0-management: ping timeout is 0, returning | |
| [2018-01-22 04:41:06.098248] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk | |
| [2018-01-22 04:41:06.098222] D [MSGID: 0] [glusterd-peer-utils.c:643:gd_add_friend_to_dict] 0-management: Returning 0 | |
| [2018-01-22 04:41:06.098248] D [MSGID: 0] [glusterd-rpc-ops.c:1730:glusterd_rpc_friend_update] 0-glusterd: Returning 0 | |
| [2018-01-22 04:41:06.098259] D [MSGID: 0] [glusterd-sm.c:690:glusterd_ac_update_friend] 0-management: Returning with 0 | |
| [2018-01-22 04:41:06.098265] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:06.098270] D [MSGID: 0] [glusterd-utils.c:7545:glusterd_sm_tr_log_transition_add] 0-management: Transitioning from 'Peer in Cluster' to 'Peer in Cluster' due to event 'GD_FRIEND_EVENT_RCVD_ACC' | |
| [2018-01-22 04:41:06.098274] D [MSGID: 0] [glusterd-utils.c:7547:glusterd_sm_tr_log_transition_add] 0-management: returning 0 | |
| [2018-01-22 04:41:06.098277] D [MSGID: 0] [glusterd-peer-utils.c:199:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster | |
| [2018-01-22 04:41:06.098287] D [MSGID: 0] [glusterd-store.c:4147:glusterd_store_create_peer_dir] 0-glusterd: Returning with 0 | |
| [2018-01-22 04:41:06.098316] D [MSGID: 0] [store.c:378:gf_store_save_value] 0-management: returning: 0 | |
| [2018-01-22 04:41:06.098330] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk | |
| The message "D [MSGID: 0] [store.c:378:gf_store_save_value] 0-management: returning: 0" repeated 2 times between [2018-01-22 04:41:06.098316] and [2018-01-22 04:41:06.098329] | |
| [2018-01-22 04:41:06.098330] D [MSGID: 0] [glusterd-store.c:4276:glusterd_store_peer_write] 0-glusterd: Returning with 0 | |
| [2018-01-22 04:41:06.098665] D [socket.c:348:ssl_setup_connection] 0-socket.management: peer CN = gluster-server | |
| [2018-01-22 04:41:06.098674] D [socket.c:351:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: 10.0.0.2:49150) (server: 10.0.0.2:24007) | |
| [2018-01-22 04:41:06.101491] D [socket.c:348:ssl_setup_connection] 0-socket.management: peer CN = gluster-client | |
| [2018-01-22 04:41:06.101502] D [socket.c:351:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: 10.0.2.2:49150) (server: 10.0.0.2:24007) | |
| [2018-01-22 04:41:06.103800] D [socket.c:348:ssl_setup_connection] 0-socket.management: peer CN = gluster-server | |
| [2018-01-22 04:41:06.103811] D [socket.c:351:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: 10.0.0.3:49149) (server: 10.0.0.2:24007) | |
| [2018-01-22 04:41:06.104313] D [socket.c:348:ssl_setup_connection] 0-socket.management: peer CN = gluster-server | |
| [2018-01-22 04:41:06.104322] D [socket.c:351:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: 10.0.0.3:49150) (server: 10.0.0.2:24007) | |
| [2018-01-22 04:41:06.104325] D [socket.c:348:ssl_setup_connection] 0-socket.management: peer CN = gluster-server | |
| [2018-01-22 04:41:06.104335] D [socket.c:351:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: 10.0.0.3:49151) (server: 10.0.0.2:24007) | |
| [2018-01-22 04:41:06.150522] D [MSGID: 0] [glusterd-store.c:4302:glusterd_store_perform_peer_store] 0-glusterd: Returning 0 | |
| [2018-01-22 04:41:06.150536] D [MSGID: 0] [glusterd-store.c:4323:glusterd_store_peerinfo] 0-glusterd: Returning with 0 | |
| [2018-01-22 04:41:06.150560] D [MSGID: 0] [glusterd-utils.c:341:glusterd_lock] 0-management: Cluster lock held by 980f9026-d35c-44e3-b432-5b0beee53635 | |
| [2018-01-22 04:41:06.150595] E [MSGID: 101012] [common-utils.c:3861:gf_is_service_running] 0-: Unable to read pidfile: /var/run/gluster/vols/myvol/10.0.0.2-data-glusterfs-myvol-brick1-brick.pid | |
| [2018-01-22 04:41:06.150622] I [glusterd-utils.c:5926:glusterd_brick_start] 0-management: starting a fresh brick process for brick /data/glusterfs/myvol/brick1/brick |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment