Skip to content

Instantly share code, notes, and snippets.

@tanabarr

tanabarr/messages3.txt

Created June 30, 2017 10:43
Show Gist options
  • Save tanabarr/9f54584621aacfdeb3899f59687cb918 to your computer and use it in GitHub Desktop.
Save tanabarr/9f54584621aacfdeb3899f59687cb918 to your computer and use it in GitHub Desktop.
Download ZIP
kernel messages showing LustreErrors related to DNE failed mount of MDT
Raw
messages3.txt
Jun 29 08:14:05 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:05 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:05 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:05 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sde): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:06 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdb): file extents enabled, maximum tree depth=5
Jun 29 08:14:06 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:12 lotus-32vm7.lotus.hpdd.lab.intel.com dhclient[674]: DHCPREQUEST on eth0 to 192.168.0.253 port 67 (xid=0x5b13ae41)
Jun 29 08:14:14 lotus-32vm7.lotus.hpdd.lab.intel.com stonith-ng[29547]: notice: On loss of CCM Quorum: Ignore
Jun 29 08:14:15 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: Result of probe operation for MGS_cbff7c on lotus-32vm7.lotus.hpdd.lab.intel.com: 7 (not running)
Jun 29 08:14:18 lotus-32vm7.lotus.hpdd.lab.intel.com stonith-ng[29547]: notice: On loss of CCM Quorum: Ignore
Jun 29 08:14:20 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sda): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:20 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sda): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 29 08:14:20 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: 13102:0:(osd_handler.c:7007:osd_mount()) MGS-osd: device /dev/sda was upgraded from Lustre-1.x without enabling the dirdata feature. If you do not want to downgrade to Lustre-1.x again, you can enable it via 'tune2fs -O dirdata device'
Jun 29 08:14:20 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: MGS: Connection restored to 344f93df-9bc4-3fd3-f3d6-9bc85bd70923 (at 0@lo)
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ [ ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ { ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "args": [ ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "modprobe", ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "osd_ldiskfs" ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ ], ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "rc": 0, ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "stderr": "", ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "stdout": "" ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ }, ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ { ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "args": [ ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "mount", ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "-t", ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "lustre", ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1", ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "/mnt/MGS" ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ ], ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "rc": 0, ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "stderr": "mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 512 to 16384\n", ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ "stdout": "" ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ } ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ ] ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: MGS_cbff7c_start_0:13081:stderr [ ]
Jun 29 08:14:21 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: Result of start operation for MGS_cbff7c on lotus-32vm7.lotus.hpdd.lab.intel.com: 0 (ok)
Jun 29 08:14:24 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: sda: WRITE SAME failed. Manually zeroing.
Jun 29 08:14:25 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:25 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:25 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdd): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 29 08:14:25 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 29 08:14:25 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: srv-fs1-MDT0001: No data found on store. Initialize space
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0001: new disk, initializing
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0001: Imperative Recovery not enabled, recovery window 300-900
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13253:0:(fid_handler.c:329:__seq_server_alloc_meta()) srv-fs1-MDT0001: Allocated super-sequence failed: rc = -115
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13253:0:(fid_request.c:227:seq_client_alloc_seq()) cli-fs1-MDT0001: Can't allocate new meta-sequence,rc -115
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13253:0:(fid_request.c:383:seq_client_alloc_fid()) cli-fs1-MDT0001: Can't allocate new sequence: rc = -115
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13253:0:(lod_dev.c:419:lod_sub_recovery_thread()) fs1-MDT0001-osd getting update log failed: rc = -115
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0001: Connection restored to 344f93df-9bc4-3fd3-f3d6-9bc85bd70923 (at 0@lo)
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sde): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sde): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: sdc: WRITE SAME failed. Manually zeroing.
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: ctl-fs1-MDT0000: No data found on store. Initialize space
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 1 previous similar message
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0000: new disk, initializing
Jun 29 08:14:26 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 1 previous similar message
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: sdd: WRITE SAME failed. Manually zeroing.
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: Failing over fs1-MDT0002
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 11-0: fs1-MDT0001-osp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13281:0:(fid_request.c:227:seq_client_alloc_seq()) cli-cli-fs1-MDT0001-osp-MDT0002: Can't allocate new meta-sequence,rc -5
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13281:0:(fid_request.c:227:seq_client_alloc_seq()) Skipped 1 previous similar message
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13281:0:(fid_request.c:383:seq_client_alloc_fid()) cli-cli-fs1-MDT0001-osp-MDT0002: Can't allocate new sequence: rc = -5
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13281:0:(fid_request.c:383:seq_client_alloc_fid()) Skipped 1 previous similar message
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13281:0:(lod_dev.c:419:lod_sub_recovery_thread()) fs1-MDT0001-osp-MDT0002 getting update log failed: rc = -5
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13281:0:(lod_dev.c:419:lod_sub_recovery_thread()) Skipped 1 previous similar message
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 137-5: fs1-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0002: Not available for connect from 0@lo (stopping)
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0000: Imperative Recovery not enabled, recovery window 300-900
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 1 previous similar message
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: ctl-fs1-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
Jun 29 08:14:27 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: server umount fs1-MDT0001 complete
Jun 29 08:14:28 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: Failing over fs1-MDT0000
Jun 29 08:14:28 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 1 previous similar message
Jun 29 08:14:28 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13312:0:(osp_object.c:582:osp_attr_get()) fs1-MDT0001-osp-MDT0000:osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5
Jun 29 08:14:28 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 13312:0:(lod_sub_object.c:932:lod_sub_prep_llog()) fs1-MDT0000-mdtlov: can't get id from catalogs: rc = -5
Jun 29 08:14:28 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: server umount fs1-MDT0000 complete
Jun 29 08:14:28 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 1 previous similar message
Jun 29 08:14:33 lotus-32vm7.lotus.hpdd.lab.intel.com stonith-ng[29547]: notice: On loss of CCM Quorum: Ignore
Jun 29 08:14:33 lotus-32vm7.lotus.hpdd.lab.intel.com dhclient[674]: DHCPREQUEST on eth0 to 192.168.0.253 port 67 (xid=0x5b13ae41)
Jun 29 08:14:33 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: Result of probe operation for fs1-MDT0001_4bd883 on lotus-32vm7.lotus.hpdd.lab.intel.com: 7 (not running)
Jun 29 08:14:33 lotus-32vm7.lotus.hpdd.lab.intel.com stonith-ng[29547]: notice: On loss of CCM Quorum: Ignore
Jun 29 08:14:34 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: Result of probe operation for fs1-MDT0002_9dce20 on lotus-32vm7.lotus.hpdd.lab.intel.com: 7 (not running)
Jun 29 08:14:34 lotus-32vm7.lotus.hpdd.lab.intel.com stonith-ng[29547]: notice: On loss of CCM Quorum: Ignore
Jun 29 08:14:34 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: Result of probe operation for fs1-MDT0000_b3ca91 on lotus-32vm7.lotus.hpdd.lab.intel.com: 7 (not running)
Jun 29 08:14:37 lotus-32vm7.lotus.hpdd.lab.intel.com stonith-ng[29547]: notice: On loss of CCM Quorum: Ignore
Jun 29 08:14:37 lotus-32vm7.lotus.hpdd.lab.intel.com stonith-ng[29547]: notice: On loss of CCM Quorum: Ignore
Jun 29 08:14:38 lotus-32vm7.lotus.hpdd.lab.intel.com stonith-ng[29547]: notice: On loss of CCM Quorum: Ignore
Jun 29 08:14:38 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdd): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 29 08:14:38 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 137-5: fs1-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jun 29 08:14:38 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0001: Imperative Recovery not enabled, recovery window 300-900
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ [ ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ { ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "args": [ ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "modprobe", ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "osd_ldiskfs" ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ ], ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "rc": 0, ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "stderr": "", ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "stdout": "" ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ }, ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ { ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "args": [ ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "mount", ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "-t", ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "lustre", ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3", ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "/mnt/fs1-MDT0001" ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ ], ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "rc": 0, ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "stderr": "", ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ "stdout": "" ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ } ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ ] ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0001_4bd883_start_0:13433:stderr [ ]
Jun 29 08:14:39 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: Result of start operation for fs1-MDT0001_4bd883 on lotus-32vm7.lotus.hpdd.lab.intel.com: 0 (ok)
Jun 29 08:14:40 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sde): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 29 08:14:40 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0001: Will be in recovery for at least 5:00, or until 1 client reconnects
Jun 29 08:14:40 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0001: Connection restored to 344f93df-9bc4-3fd3-f3d6-9bc85bd70923 (at 0@lo)
Jun 29 08:14:40 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 137-5: fs1-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jun 29 08:14:40 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: Skipped 2 previous similar messages
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ [ ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ { ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "args": [ ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "modprobe", ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "osd_ldiskfs" ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ ], ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "rc": 0, ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "stderr": "", ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "stdout": "" ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ }, ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ { ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "args": [ ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "mount", ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "-t", ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "lustre", ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2", ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "/mnt/fs1-MDT0000" ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ ], ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "rc": 0, ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "stderr": "", ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ "stdout": "" ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ } ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ ] ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0000_b3ca91_start_0:13515:stderr [ ]
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: Result of start operation for fs1-MDT0000_b3ca91 on lotus-32vm7.lotus.hpdd.lab.intel.com: 0 (ok)
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LustreError: 11-0: fs1-MDT0001-osp-MDT0002: operation mds_connect to node 0@lo failed: rc = -114
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: sde: WRITE SAME failed. Manually zeroing.
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: fs1-MDT0002: Imperative Recovery not enabled, recovery window 300-900
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 1 previous similar message
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: ctl-fs1-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:2:mdt
Jun 29 08:14:41 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: Lustre: cli-ctl-fs1-MDT0002: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:2:mdt]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ [ ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ { ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "args": [ ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "modprobe", ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "osd_ldiskfs" ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ ], ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "rc": 0, ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "stderr": "", ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "stdout": "" ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ }, ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ { ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "args": [ ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "mount", ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "-t", ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "lustre", ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4", ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "/mnt/fs1-MDT0002" ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ ], ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "rc": 0, ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "stderr": "", ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ "stdout": "" ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ } ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ ] ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com lrmd[29548]: notice: fs1-MDT0002_9dce20_start_0:13514:stderr [ ]
Jun 29 08:14:42 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: Result of start operation for fs1-MDT0002_9dce20 on lotus-32vm7.lotus.hpdd.lab.intel.com: 0 (ok)
Jun 29 08:14:43 lotus-32vm7.lotus.hpdd.lab.intel.com dhclient[674]: DHCPREQUEST on eth0 to 192.168.0.253 port 67 (xid=0x5b13ae41)
Jun 29 08:14:47 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdb): file extents enabled, maximum tree depth=5
Jun 29 08:14:47 lotus-32vm7.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Jun 29 08:14:55 lotus-32vm7.lotus.hpdd.lab.intel.com dhclient[674]: DHCPREQUEST on eth0 to 192.168.0.253 port 67 (xid=0x5b13ae41)
Jun 29 08:15:01 lotus-32vm7.lotus.hpdd.lab.intel.com crmd[29551]: notice: High CPU load detected: 3.720000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment