Skip to content

Instantly share code, notes, and snippets.

@joshuar
Last active July 22, 2020 09:52
Show Gist options
  • Save joshuar/4e283308c932ec62fc05 to your computer and use it in GitHub Desktop.
Save joshuar/4e283308c932ec62fc05 to your computer and use it in GitHub Desktop.
Quick three-node Lustre set-up on CentOS 6

Mostly following the guide found here:

https://wiki.hpdd.intel.com/display/PUB/Create+and+Mount+a+Lustre+Filesystem

  • Downgrade kernel if needed:
    • yum remove kernel-2.6.32-<whatever>*
  • Enable EPEL repo:
    • yum install epel-release
  • Enable ZFS on Linux repo:
    • yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
  • Enable lustre repos:
[lustre-server]
name=CentOS-$releasever - Lustre
baseurl=https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/
gpgcheck=0

[e2fsprogs]
name=CentOS-$releasever - Ldiskfs
baseurl=https://downloads.hpdd.intel.com/public/e2fsprogs/latest/el6/RPMS
gpgcheck=0

[lustre-client]
name=CentOS-$releasever - Lustre
baseurl=https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/client/
gpgcheck=0
  • On all of the MDS, OSS and client machines:
    • Upgrade e2fsprogs: yum upgrade e2fsprogs
    • Install the lustre-tests package: yum install lustre-tests
    • Create the following file /etc/modprobe.d/lnet.conf:
      • options lnet networks=tcp0(eth1)
      • Adjust the networks parameter as required for the type of interconnect and network interface to be used.
  • On the MGS and OSS only, create the file /etc/sysconfig/modules/lnet.modules to auto-load the lnet kernel module on boot:
#!/bin/sh

if [ ! -c /dev/lnet ] ; then
    exec /sbin/modprobe lnet >/dev/null 2>&1
fi
  • On the MGS/MDT/MDS:
    • Intialise a disk or partition to use for lustre.
    • Create a lustre MDT:
      • mkfs.lustre --fsname=whatevs --mgs --mdt --index=0 /dev/sdX
    • Create a mount point and mount the lustre FS:
      • mkdir /mnt/mdt && mount -t lustre /dev/sdX /mnt/mdt
  • On the OST/OSS:
    • Intialise a disk or partition to use for lustre.
    • Create a lustre OST:
      • mkfs.lustre --ost --fsname=whatevs --mgsnode=192.168.N.N@tcp0 --index=0 /dev/sdX
      • Adjust the --mgsnode parameter for the address and protocol used for the MGS.
    • Create a mount point and mount the lustre FS:
      • mkdir /ostoss_mount && mount -t lustre /dev/sdX /ostoss_mount
  • On the client:
    • Load the lustre kernel module: modprobe lustre.
    • Create the file /etc/sysconfig/modules/lustre.modules to load the lustre module on boot:
#!/bin/sh

/sbin/lsmod | /bin/grep lustre 1>/dev/null 2>&1
if [ ! $? ] ; then
   /sbin/modprobe lustre >/dev/null 2>&1
fi
  • Create a mount point: mkdir /mnt/lustre.
    • Mount the lustre FS:
      • mount -t lustre 192.168.N.N@tcp0:/whatevs /mnt/lustre
@jafffy
Copy link

jafffy commented Sep 21, 2015

Have you ever experienced when you mount oss it just wait and put error message once 120 seconds like
INFO: task mount.lustre:1541 blocked for more than 120 seconds.
Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
?

@oxedions
Copy link

Had the same error. You need to disable completely SELinux (not just permissive, disabled in /etc/selinux/config, then reboot). Then redo mkfs.lustre, now it should work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment