kernel messages showing LustreErrors related to DNE failed mount of MDT

messages.txt

Jun 30 01:55:03 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 30 01:55:03 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 30 01:55:03 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sde): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 30 01:55:03 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sde): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: srv-testfs-MDT0001: No data found on store. Initialize space
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: testfs-MDT0001: new disk, initializing
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: testfs-MDT0001: Imperative Recovery not enabled, recovery window 300-900
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 4407:0:(fid_handler.c:329:__seq_server_alloc_meta()) srv-testfs-MDT0001: Allocated super-sequence failed: rc = -115
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 4407:0:(fid_request.c:227:seq_client_alloc_seq()) cli-testfs-MDT0001: Can't allocate new meta-sequence,rc -115
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 4407:0:(fid_request.c:383:seq_client_alloc_fid()) cli-testfs-MDT0001: Can't allocate new sequence: rc = -115
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 4407:0:(lod_dev.c:419:lod_sub_recovery_thread()) testfs-MDT0001-osd getting update log failed: rc = -115
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: testfs-MDT0001: Connection restored to e1711fdf-6dd9-9463-7436-8edaf2bd2d56 (at 0@lo)
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: ctl-testfs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: Failing over testfs-MDT0001
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 11-0: testfs-MDT0001-osp-MDT0000: operation mds_disconnect to node 0@lo failed: rc = -107
Jun 30 01:55:04 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: server umount testfs-MDT0001 complete
Jun 30 01:55:07 lotus-52vm5.lotus.hpdd.lab.intel.com firewalld: ERROR: ALREADY_ENABLED: '988:tcp' already in 'public'
Jun 30 01:55:08 lotus-52vm5.lotus.hpdd.lab.intel.com firewalld: ERROR: ALREADY_ENABLED: 988:tcp
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com stonith-ng[3109]:  notice: On loss of CCM Quorum: Ignore
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: On loss of CCM Quorum: Ignore
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: Calculated transition 18, saving inputs in /var/lib/pacemaker/pengine/pe-input-30.bz2
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: Initiating monitor operation testfs-MDT0001_5cfc6a_monitor_0 on lotus-52vm6.lotus.hpdd.lab.intel.com
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: Initiating monitor operation testfs-MDT0001_5cfc6a_monitor_0 locally on lotus-52vm5.lotus.hpdd.lab.intel.com
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: Result of probe operation for testfs-MDT0001_5cfc6a on lotus-52vm5.lotus.hpdd.lab.intel.com: 7 (not running)
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: Transition 18 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-30.bz2): Complete
Jun 30 01:55:13 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE
Jun 30 01:55:14 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
Jun 30 01:55:14 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: On loss of CCM Quorum: Ignore
Jun 30 01:55:14 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: Calculated transition 19, saving inputs in /var/lib/pacemaker/pengine/pe-input-31.bz2
Jun 30 01:55:14 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: Transition 19 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-31.bz2): Complete
Jun 30 01:55:14 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 4425:0:(fid_request.c:227:seq_client_alloc_seq()) cli-cli-testfs-MDT0001-osp-MDT0000: Can't allocate new meta-sequence,rc -5
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 4425:0:(fid_request.c:383:seq_client_alloc_fid()) cli-cli-testfs-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 4425:0:(lod_dev.c:419:lod_sub_recovery_thread()) testfs-MDT0001-osp-MDT0000 getting update log failed: rc = -5
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: On loss of CCM Quorum: Ignore
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: Calculated transition 20, saving inputs in /var/lib/pacemaker/pengine/pe-input-32.bz2
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: Transition 20 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-32.bz2): Complete
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE
Jun 30 01:55:15 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: server umount testfs-MDT0000 complete
Jun 30 01:55:17 lotus-52vm5.lotus.hpdd.lab.intel.com stonith-ng[3109]:  notice: On loss of CCM Quorum: Ignore
Jun 30 01:55:17 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
Jun 30 01:55:17 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: On loss of CCM Quorum: Ignore
Jun 30 01:55:17 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: Start   testfs-MDT0001_5cfc6a#011(lotus-52vm5.lotus.hpdd.lab.intel.com)
Jun 30 01:55:17 lotus-52vm5.lotus.hpdd.lab.intel.com pengine[3112]:  notice: Calculated transition 21, saving inputs in /var/lib/pacemaker/pengine/pe-input-33.bz2
Jun 30 01:55:17 lotus-52vm5.lotus.hpdd.lab.intel.com crmd[3113]:  notice: Initiating start operation testfs-MDT0001_5cfc6a_start_0 locally on lotus-52vm5.lotus.hpdd.lab.intel.com
Jun 30 01:55:18 lotus-52vm5.lotus.hpdd.lab.intel.com firewalld: ERROR: ALREADY_ENABLED: '988:tcp' already in 'public'
Jun 30 01:55:18 lotus-52vm5.lotus.hpdd.lab.intel.com firewalld: ERROR: ALREADY_ENABLED: 988:tcp
Jun 30 01:55:19 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 30 01:55:19 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: LustreError: 137-5: testfs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jun 30 01:55:19 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: testfs-MDT0001: Imperative Recovery not enabled, recovery window 300-900
Jun 30 01:55:19 lotus-52vm5.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 1 previous similar message