Skip to content

Instantly share code, notes, and snippets.

@RulerOf

RulerOf/creating-zfs-pool-on-zol.md

Last active October 7, 2020 01:30
Show Gist options
  • Save RulerOf/e1e7f4a6b56533dcb7822d4b9ce58e3a to your computer and use it in GitHub Desktop.
Save RulerOf/e1e7f4a6b56533dcb7822d4b9ce58e3a to your computer and use it in GitHub Desktop.
Download ZIP
Setting up a 20-disk raidz3 with Fusion io ZIL and cache with ZoL on Oracle Linux 7
Raw
creating-zfs-pool-on-zol.md

Creating a raidz3 pool with Fusion io SLOG and L2ARC cache

Using the following hardware:

  • 23 3TB Hitachi Ultrastar disks
  • 2 640GB Fusion io ioDrive Duo cards, each with 2 separate storage devices on them

The plan is to create a RAIDz3 zpool with 20 disks in the array, 3 hot spares, a mirrored log device containing one drive from each of the ioDrives, and two cache devices made out of the remaining ioDrives. We'll also underprovision the ioDrives to help with wear leveling, since 320GB ZIL and 640GB cache are excessive for this setup.

Underprovision the ioDrives

Detach the drives

[root@drew-metal ~]# ls /dev/fct* | xargs fio-detach
 \ Detaching - /dev/fct0: 100% | /dev/fct1: 100% | /dev/fct2: 100% | /dev/fct3: 100%
 fioa - detached.
 fiob - detached.
 fioc - detached.
 fiod - detached.

Format at 50% capacity

[root@drew-metal ~]# ls /dev/fct* | xargs fio-format --device-size 50% -y

/dev/fct0: Creating block device.
  Block device of size 160.00GBytes (149.01GiBytes).
  Using default block (sector) size of 512 bytes.

/dev/fct1: Creating block device.
  Block device of size 160.00GBytes (149.01GiBytes).
  Using block (sector) size of 512 bytes.

/dev/fct2: Creating block device.
  Block device of size 160.00GBytes (149.01GiBytes).
  Using block (sector) size of 512 bytes.

/dev/fct3: Creating block device.
  Block device of size 160.00GBytes (149.01GiBytes).
  Using block (sector) size of 512 bytes.

WARNING: Do not interrupt the formatting! If interrupted, the fio-sure-erase utility may help recover from format errors. Please see documentation or contact support.

 / Formatting - /dev/fct0: 100% | /dev/fct1: 100% | /dev/fct2: 100% | /dev/fct3: 100%
 /dev/fct0 - format successful.
 /dev/fct1 - format successful.
 /dev/fct2 - format successful.
 /dev/fct3 - format successful.

Reattach

[root@drew-metal ~]# ls /dev/fct* | xargs fio-attach
 - Attaching - /dev/fct0: 100% | /dev/fct1: 100% | /dev/fct2: 100% | /dev/fct3: 100%
 fioa - attached.
 fiob - attached.
 fioc - attached.
 fiod - attached.

Create the zpool

We're going to call this pool storage. I've disconnected the three disks that will act as hot spares and will add them to the array separately.

Create the array with SLOG and Cache, setting some ZFS defaults that I've worked out:

[root@drew-metal ~]# DISKS=$(find /dev/disk/by-id/ -name '*HITACHI*' ! -name '*part*' -exec basename {} \;)
[root@drew-metal ~]# zpool create -o autoreplace=on -o autoexpand=on -O atime=off -O relatime=on -O compress=lz4 -O dnodesize=auto -O xattr=sa storage raidz3 $DISKS log mirror fioa fioc cache fiob fiod

Check the array's status to verify that it came out the way we want it

[root@drew-metal ~]# zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

	NAME                                      STATE     READ WRITE CKSUM
	storage                                   ONLINE       0     0     0
	  raidz3-0                                ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHDBSTA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHH40RHA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHD36ZA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHH9K8RA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHD7A0A  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHDVZ9A  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHH7H9UA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHDBDWA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHDUXTA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHE7H9A  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHAP1PA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHH7HGPA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHDKU7A  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHE3XRA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHE5NZA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHH948XA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHE085A  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHDEEBA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHASBTA  ONLINE       0     0     0
	    ata-HITACHI_HUA723030ALA640_YHHE4Y7A  ONLINE       0     0     0
	logs
	  mirror-1                                ONLINE       0     0     0
	    fioa                                  ONLINE       0     0     0
	    fioc                                  ONLINE       0     0     0
	cache
	  fiob                                    ONLINE       0     0     0
	  fiod                                    ONLINE       0     0     0

errors: No known data errors
[root@drew-metal ~]#

Create some data sets

One dataset for media storage, and another for VM zvols

zfs create storage/media
zfs create -o canmount=noauto storage/vm

Examples for VM storage

Create VM ZVOLs like this:

zfs create -b 8k -V 10G storage/vm/myLinuxVM

Create deduped VM ZVOLs like this:

zfs create -b 8k -s -o dedup=on -V 10G storage/vm/myWin10VM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment