Skip to content

Instantly share code, notes, and snippets.

@patrickmslatteryvt

patrickmslatteryvt/ZFS on Linux (CentOS v6.5)

Last active August 29, 2015 14:00
Show Gist options
  • Save patrickmslatteryvt/11049118 to your computer and use it in GitHub Desktop.
Save patrickmslatteryvt/11049118 to your computer and use it in GitHub Desktop.
Download ZIP
ZFS on Linux (CentOS v6.5)
Raw
ZFS on Linux (CentOS v6.5)
ZFS on Linux
FAQ
http://zfsonlinux.org/faq.html
A not so short guide to ZFS on Linux
http://unicolet.blogspot.com/2013/03/a-not-so-short-guide-to-zfs-on-linux.html
ZFS Cheatsheet
http://www.datadisk.co.uk/html_docs/sun/sun_zfs_cs.htm
http://www.zfsbuild.com/
Disk layout in a test VM:
SCSI 0:0 = 10GB HDD on paravirtual, would be equivalent to RAIDed Mirror in physical system
This will become:
/boot
swap
/
SCSI 1:0 = 16GB HDD on LSI SAS
SCSI 1:1 = 16GB HDD on LSI SAS
SCSI 1:2 = 16GB HDD on LSI SAS
SCSI 1:3 = 16GB HDD on LSI SAS
SCSI 1:4 = 16GB HDD on LSI SAS
SCSI 1:5 = 16GB HDD on LSI SAS
SCSI 1:6 = 16GB HDD on LSI SAS
SCSI 1:8 = 16GB HDD on LSI SAS
This will become a mirrored vdev pool mounted at /srv
SCSI 2:0 = 14GB SSD on LSI SAS
SCSI 2:1 = 14GB SSD on LSI SAS
This will become the ZIL (2GB) and L2ARC (12GB)
Faking an SSD in a vSphere VM
http://www.vladan.fr/vmware-vsan-configuration/
Add to Configuration Parameters
scsi2:0.virtualSSD = 1
scsi2:1.virtualSSD = 1
Enable UUIDs for disks in a vSphere VM
http://www.novell.com/support/kb/doc.php?id=7002966
Add to Configuration Parameters
disk.EnableUUID = TRUE
Restart the vm machine and the /dev/disk/by-id/scsi**** should be visible.
/dev/disk/by-path/: Good for large pools (greater than 10 disks)
Summary: This approach is to use device names which include the physical cable layout in the system, which means that a particular disk is tied to a specific location. The name describes the PCI bus number, as well as enclosure names and port numbers. This allows the most control when configuring a large pool.
Benefits: Encoding the storage topology in the name is not only helpful for locating a disk in large installations. But it also allows you to explicitly layout your redundancy groups over multiple adapters or enclosures.
Drawbacks: These names are long, cumbersome, and difficult for a human to manage.
Example:
$ sudo zpool create tank pci-0000:00:1f.2-scsi-0:0:0:0 pci-0000:00:1f.2-scsi-1:0:0:0
Also checkout:
/dev/disk/by-vdev/: Best for large pools (greater than 10 disks)
Initial - before OS install
Patrick M. Slattery
April 17th 2014
boot from CentOS v6.5 network install ISO with boot option:
... ks=http://kicker/ks/zol.ks
# Must be installed before we install ZFS
yum install -y kernel
# Does not seem to provide the headers we need
http://zfsonlinux.org/epel.html
yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release-1-3.el6.noarch.rpm
yum install -y zfs
Loading new spl-0.6.2 DKMS files...
First Installation: checking all kernels...
Building only for 2.6.32-431.el6.x86_64
Module build for the currently running kernel was skipped since the kernel source for this kernel does not seem to be installed.
Installing : zfs-dkms-0.6.2-1.el6.noarch 13/15
Loading new zfs-0.6.2 DKMS files...
First Installation: checking all kernels...
Building only for 2.6.32-431.el6.x86_64
Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed.
yum install kernel-devel # Provides /usr/src/kernels/2.6.32-431.11.2.el6.x86_64
First Installation: checking all kernels...
Building only for 2.6.32-431.11.2.el6.x86_64
Building initial module for 2.6.32-431.11.2.el6.x86_64
Done.
spl:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/2.6.32-431.11.2.el6.x86_64/extra/
splat.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/2.6.32-431.11.2.el6.x86_64/extra/
Adding any weak-modules
Running the post_install script:
# unlink /lib/modules/$(uname -r)/build
# ln -s /usr/src/kernels/$(uname -r) /lib/modules/$(uname -r)/build
# /usr/src/kernels/2.6.32-431.11.2.el6.x86_64/
# build -> /usr/src/kernels/linux-2.6.32-431.el6.x86_64
lsmod | grep zfs
[root@smartos01 ~]# ls -la /dev/disk/by-path/|grep sas
HDD
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0 -> ../../sdi
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0 -> ../../sdh
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0 -> ../../sdg
SSD
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0 -> ../../sdj
lrwxrwxrwx. 1 root root 9 Apr 21 19:58 pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0 -> ../../sdk
pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0 pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0 pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0 pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0 pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0 pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0 pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0 pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0
pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0
pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0
# Main pool
# Dry run
zpool create -n -f svnpool mirror pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0 pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0 pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0 pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0 pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0 pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0 pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0 pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0 -m /srv
# Do it
zpool create -f svnpool mirror pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0 pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0 pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0 pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0 pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0 pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0 pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0 pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0 -m /srv
# ZIL + L2ARC
# Partitioning - ZIL (2GB) and L2ARC (12GB)
parted /dev/sdi mklabel gpt
parted /dev/sdj mklabel gpt
parted /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0 mklabel gpt
parted /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0 mklabel gpt
# ZIL
parted /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0 mkpart primary 0% 15%
parted /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0 mkpart primary 0% 15%
# L2ARC
parted /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0 mkpart primary 15% 100%
parted /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0 mkpart primary 15% 100%
parted -l
Model: VMware Virtual disk (scsi)
Disk /dev/sdk: 15.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 2254MB 2253MB primary
2 2254MB 15.0GB 12.8GB primary
Model: VMware Virtual disk (scsi)
Disk /dev/sdj: 15.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 2254MB 2253MB primary
2 2254MB 15.0GB 12.8GB primary
[root@smartos01 ~]# ls -la /dev/disk/by-path/|grep 0x5000c29416cb8f9f
lrwxrwxrwx. 1 root root 9 Apr 21 20:27 pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Apr 21 20:27 pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Apr 21 20:27 pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0-part2 -> ../../sdj2
# Add a mirrored ZIL
zpool add svnpool log mirror /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0-part1 /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0-part1
# Add L2ARC devices (need need to be mirrored, L2ARC will be ignored if a device fails - at the cost of cache hits dropping to 0%)
zpool add svnpool cache /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0-part2
zpool add svnpool cache /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0-part2
[root@smartos01 ~]# zpool status
pool: svnpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
svnpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0 ONLINE 0 0 0
pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0 ONLINE 0 0 0
pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0 ONLINE 0 0 0
pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0 ONLINE 0 0 0
pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0 ONLINE 0 0 0
pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0 ONLINE 0 0 0
pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0 ONLINE 0 0 0
pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0-part1 ONLINE 0 0 0
pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0-part1 ONLINE 0 0 0
cache
pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0-part2 ONLINE 0 0 0
pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0-part2 ONLINE 0 0 0
errors: No known data errors
# Use zilstat for monitoring ZIL performance.
http://www.andybotting.com/zfs-on-linux
# Turn off access time updates
zfs set atime=off svnpool
# Turn off dedupe
zfs set dedup=off svnpool
Create our documents filesystem, mount and share it by NFS
# zfs create mypool/documents
# zfs set mountpoint=/mnt/documents mypool/documents
# zfs set sharenfs=on mypool/documents
Create our photos filesystem, mount and share it by NFS
# zfs create mypool/photos
# zfs set mountpoint=/mnt/photos mypool/photos
# zfs set sharenfs=on mypool/photos
Photos are important, so keep two copies of them around
# zfs set copies=2 mypool/photos
Documents are really important, so we’ll keep three copies of them on disk
# zfs set copies=3 mypool/documents
Documents are mostly text, so we’ll compress them.
zfs set compression=on mypool/documents
ZFS pools should be scrubbed at least once a week. It helps balance the data across the disks in your pool and to fix up any data integrity errors it might find.
# zpool scrub <pool>
To do automatic scrubbing once a week, set the following line in your root crontab
# crontab -e
...
30 19 * * 5 zpool scrub <pool>
...
#==============================================================================
# nano -w /etc/zfs/vdev_id.conf
# by-vdev
# name fully qualified or base name of device link
# HDDs
alias c1t0d0 /dev/disk/by-path/pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0
alias c1t1d0 /dev/disk/by-path/pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0
alias c1t2d0 /dev/disk/by-path/pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0
alias c1t3d0 /dev/disk/by-path/pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0
alias c1t4d0 /dev/disk/by-path/pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0
alias c1t5d0 /dev/disk/by-path/pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0
alias c1t6d0 /dev/disk/by-path/pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0
alias c1t8d0 /dev/disk/by-path/pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0
# SSDs
alias c2t0d0 /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0
alias c2t1d0 /dev/disk/by-path/pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0
udevadm trigger
[root@smartos01 ~]# ls -la /dev/disk/by-vdev
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c1t0d0 -> ../../sdi
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c1t1d0 -> ../../sdf
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c1t2d0 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c1t3d0 -> ../../sdd
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c1t4d0 -> ../../sdc
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c1t5d0 -> ../../sdh
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c1t6d0 -> ../../sde
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c1t8d0 -> ../../sdg
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c2t0d0 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Apr 21 20:56 c2t0d0-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Apr 21 20:56 c2t0d0-part2 -> ../../sdj2
lrwxrwxrwx. 1 root root 9 Apr 21 20:56 c2t1d0 -> ../../sdk
lrwxrwxrwx. 1 root root 10 Apr 21 20:56 c2t1d0-part1 -> ../../sdk1
lrwxrwxrwx. 1 root root 10 Apr 21 20:56 c2t1d0-part2 -> ../../sdk2
zpool create -f svnpool mirror c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0 c1t8d0 -m /srv
# Add a mirrored ZIL
zpool add svnpool log mirror c2t0d0-part1 c2t1d0-part1
# Add L2ARC devices (need need to be mirrored, L2ARC will be ignored if a device fails - at the cost of cache hits dropping to 0%)
zpool add svnpool cache c2t0d0-part2
zpool add svnpool cache c2t1d0-part2
[root@smartos01 ~]# zpool status
pool: svnpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
svnpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c1t6d0 ONLINE 0 0 0
c1t8d0 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
c2t0d0-part1 ONLINE 0 0 0
c2t1d0-part1 ONLINE 0 0 0
cache
c2t0d0-part2 ONLINE 0 0 0
c2t1d0-part2 ONLINE 0 0 0
errors: No known data errors
#==============================================================================
# What happens if you don't use disk by ID or similar
[root@smartos01 ~]# zpool status -x
pool: svnpool
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool to continue
functioning.
action: Destroy and re-create the pool from
a backup source.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
scan: none requested
config:
NAME STATE READ WRITE CKSUM
svnpool UNAVAIL 0 0 0 insufficient replicas
mirror-0 UNAVAIL 0 0 0 insufficient replicas
sda UNAVAIL 0 0 0
sdb FAULTED 0 0 0 corrupted data
sdc FAULTED 0 0 0 corrupted data
sdd FAULTED 0 0 0 corrupted data
sde FAULTED 0 0 0 corrupted data
sdf FAULTED 0 0 0 corrupted data
sdg FAULTED 0 0 0 corrupted data
sdh FAULTED 0 0 0 corrupted data
logs
mirror-1 UNAVAIL 0 0 0 insufficient replicas
sdi1 FAULTED 0 0 0 corrupted data
sdj1 FAULTED 0 0 0 corrupted data
[root@smartos01 ~]# ls -la /dev/disk/by-path/
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:00:07.1-scsi-1:0:0:0 -> ../../sr0
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sdk
lrwxrwxrwx. 1 root root 10 Apr 17 20:48 pci-0000:03:00.0-scsi-0:0:0:0-part1 -> ../../sdk1
lrwxrwxrwx. 1 root root 10 Apr 17 20:48 pci-0000:03:00.0-scsi-0:0:0:0-part2 -> ../../sdk2
lrwxrwxrwx. 1 root root 10 Apr 17 20:48 pci-0000:03:00.0-scsi-0:0:0:0-part3 -> ../../sdk3
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0 -> ../../sdh
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0 -> ../../sdg
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0 -> ../../sdi
lrwxrwxrwx. 1 root root 9 Apr 17 20:48 pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0 -> ../../sdj
[root@smartos01 ~]# ls -la /dev/disk/by-path/
total 0
drwxr-xr-x. 2 root root 740 Apr 18 17:13 .
drwxr-xr-x. 5 root root 100 Apr 18 17:13 ..
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:00:07.1-scsi-1:0:0:0 -> ../../sr0
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:03:00.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:03:00.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:03:00.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0 -> ../../sdi
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0-part1 -> ../../sdi1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c2972c554bd9-lun-0-part9 -> ../../sdi9
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0-part1 -> ../../sdf1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c299384ccb44-lun-0-part9 -> ../../sdf9
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29b0300ef9c-lun-0-part9 -> ../../sdb9
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0-part1 -> ../../sdd1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29bdcfdea0d-lun-0-part9 -> ../../sdd9
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29bf0e1dea3-lun-0-part9 -> ../../sdc9
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0 -> ../../sdh
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0-part1 -> ../../sdh1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29cff6fba10-lun-0-part9 -> ../../sdh9
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0-part1 -> ../../sde1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29dc4c92e62-lun-0-part9 -> ../../sde9
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0 -> ../../sdg
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0-part1 -> ../../sdg1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:0b:00.0-sas-0x5000c29f255ff427-lun-0-part9 -> ../../sdg9
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:13:00.0-sas-0x5000c29416cb8f9f-lun-0-part2 -> ../../sdj2
lrwxrwxrwx. 1 root root 9 Apr 18 17:13 pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0 -> ../../sdk
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0-part1 -> ../../sdk1
lrwxrwxrwx. 1 root root 10 Apr 18 17:13 pci-0000:13:00.0-sas-0x5000c29c1ba8232d-lun-0-part2 -> ../../sdk2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment