Skip to content

Instantly share code, notes, and snippets.

@dlangille
Last active November 1, 2018 22:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dlangille/be9bea06795fe60579a648863a75f659 to your computer and use it in GitHub Desktop.
Save dlangille/be9bea06795fe60579a648863a75f659 to your computer and use it in GitHub Desktop.
replacing a failing drive with a new drive
I've moved the failed drive from a drive caddy into the interior of the chassis. It is just sitting there, loose.
[dan@knew:~] $ zpool status system
pool: system
state: ONLINE
scan: scrub repaired 0 in 21h45m with 0 errors on Thu Nov 1 01:31:14 2018
config:
NAME STATE READ WRITE CKSUM
system ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da3p3 ONLINE 0 0 0
da10p3 ONLINE 0 0 0
da15p3 ONLINE 0 0 0
da4p3 ONLINE 0 0 0
da13p3 ONLINE 0 0 0
da12p3 ONLINE 0 0 0
da9p3 ONLINE 0 0 0
da14p3 ONLINE 0 0 0
da11p3 ONLINE 0 0 0
da5p3 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
da6p1 ONLINE 0 0 0
da18p1 ONLINE 0 0 0
da16p1 ONLINE 0 0 0
da8p1 ONLINE 0 0 0
da0p1 ONLINE 0 0 0
da1p1 ONLINE 0 0 0
da17p1 ONLINE 0 0 0
da19p1 ONLINE 0 0 0
da2p1 ONLINE 0 0 0
da20p1 ONLINE 0 0 0
errors: No known data errors
[dan@knew:~] $
From https://twitter.com/DLangille/status/1057724325489446912) I knew the drive being replaced has serial number 653DK7WAFS9A
Scrolling through /var/run/dmesg.boot, I found the drive here:
da18: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SPC-4 SCSI device
da18: Serial Number 653DK7WAFS9A
da18: 600.000MB/s transfers
da18: Command Queueing enabled
da18: 4769307MB (9767541168 512 byte sectors)
sas2ircu can also display drive information, and much more
[dan@knew:~] $ sudo sas2ircu list
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.
Adapter Vendor Device SubSys SubSys
Index Type ID ID Pci Address Ven ID Dev ID
----- ------------ ------ ------ ----------------- ------ ------
0 SAS2008 1000h 72h 00h:02h:00h:00h 1000h 3020h
Adapter Vendor Device SubSys SubSys
Index Type ID ID Pci Address Ven ID Dev ID
----- ------------ ------ ------ ----------------- ------ ------
1 SAS2008 1000h 72h 00h:03h:00h:00h 1000h 3020h
Adapter Vendor Device SubSys SubSys
Index Type ID ID Pci Address Ven ID Dev ID
----- ------------ ------ ------ ----------------- ------ ------
2 SAS2008 1000h 72h 00h:04h:00h:00h 1000h 3020h
SAS2IRCU: Utility Completed Successfully.
[dan@knew:~] $ sudo sas2ircu 0 display
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.
Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS2008
BIOS version : 7.39.00.00
Firmware version : 20.00.04.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 255
Concurrent commands supported : 3432
Slot : 238
Segment : 0
Bus : 2
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0
Device is a Hard disk
Enclosure # : 1
Slot # : 0
SAS Address : 4433221-1-0300-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : X6IEKELNF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 1
SAS Address : 4433221-1-0200-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : 579IK5RMF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 2
SAS Address : 4433221-1-0100-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653BK12JFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 3
SAS Address : 4433221-1-0000-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653DK7WCFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 4
SAS Address : 4433221-1-0700-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : X643KHBFF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 5
SAS Address : 4433221-1-0600-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : 37D4KBJPF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 6
SAS Address : 4433221-1-0500-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653EK93QFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 7
SAS Address : 4433221-1-0400-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653IK1IBFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
Enclosure# : 1
Logical ID : 500605b0:03441a70
Numslots : 8
StartSlot : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.
[dan@knew:~] $ sudo sas2ircu 1 display
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.
Read configuration has been initiated for controller 1
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS2008
BIOS version : 7.39.00.00
Firmware version : 20.00.04.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 255
Concurrent commands supported : 3432
Slot : 6
Segment : 0
Bus : 3
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0
Device is a Hard disk
Enclosure # : 1
Slot # : 0
SAS Address : 4433221-1-0300-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653DK7WPFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 1
SAS Address : 4433221-1-0200-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : 579HKDZYF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 2
SAS Address : 4433221-1-0100-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : 4728K24SF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 3
SAS Address : 4433221-1-0000-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : 5782KL6MF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 4
SAS Address : 4433221-1-0700-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 6525K2DGFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 5
SAS Address : 4433221-1-0600-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : 579HKDZXF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 6
SAS Address : 4433221-1-0500-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : 5782KL6VF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 7
SAS Address : 4433221-1-0400-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA HDWE150
Firmware Revision : FP2A
Serial No : 37KVK1JRF57D
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
Enclosure# : 1
Logical ID : 500605b0:0228fde0
Numslots : 8
StartSlot : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.
[dan@knew:~] $ sudo sas2ircu 2 display
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.
Read configuration has been initiated for controller 2
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS2008
BIOS version : 7.39.00.00
Firmware version : 20.00.04.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 255
Concurrent commands supported : 3432
Slot : 238
Segment : 0
Bus : 4
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0
Device is a Hard disk
Enclosure # : 1
Slot # : 0
SAS Address : 4433221-1-0300-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653EK93PFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 1
SAS Address : 4433221-1-0200-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653BK12FFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 2
SAS Address : 4433221-1-0100-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653AK2MXFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 3
SAS Address : 4433221-1-0000-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 6539K3OJFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
Device is a Hard disk
Enclosure # : 1
Slot # : 5
SAS Address : 4433221-1-0600-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 4769307/9767541167
Manufacturer : ATA
Model Number : TOSHIBA MD04ACA5
Firmware Revision : FP2A
Serial No : 653DK7WAFS9A
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
Enclosure# : 1
Logical ID : 500605b0:03e8e5f0
Numslots : 8
StartSlot : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.
Since I moved /dev/da7 to a different connection on an HBA, I would expect the new drive to be at da7 now, and it is:
[dan@knew:~] $ sudo smartctl -i /dev/da7
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba X300
Device Model: TOSHIBA HDWE150
Serial Number: 579IK5RMF57D
LU WWN Device Id: 5 000039 7cbf8096f
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Nov 1 16:19:25 2018 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
[dan@knew:~] $
I confirmed this serial number via the photgraphs I took during the replacement process.
The old drive (identifed above via /var/run/dmesg.boot) is da18 and looks like this:
[dan@knew:~] $ gpart show da18
=> 34 9767541101 da18 GPT (4.5T)
34 6 - free - (3.0K)
40 9766000000 1 freebsd-zfs (4.5T)
9766000040 1541095 - free - (752M)
[dan@knew:~] $
[dan@knew:~] $ gpart show da7
gpart: No such geom: da7.
[dan@knew:~] $ sudo gpart create -s gpt da7
da7 created
[dan@knew:~] $ sudo gpart add -t freebsd-zfs -a 4K -s 9766000000 -l 579IK5RMF57D da7
da7p1 added
[dan@knew:~] $ gpart show da7
=> 40 9767541088 da7 GPT (4.5T)
40 9766000000 1 freebsd-zfs (4.5T)
9766000040 1541088 - free - (752M)
[dan@knew:~] $
These are different drive models. The original (30) is a 512-byte sector drive. The new (40) one is a 4K sector drive.
All good there.
[dan@knew:~] $ sudo zpool replace system da18p1 da7p1
[dan@knew:~] $ zpool status system
pool: system
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Nov 1 16:37:03 2018
1.07G scanned out of 44.3T at 138M/s, 93h50m to go
19.1M resilvered, 0.00% done
config:
NAME STATE READ WRITE CKSUM
system ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da3p3 ONLINE 0 0 0
da10p3 ONLINE 0 0 0
da15p3 ONLINE 0 0 0
da4p3 ONLINE 0 0 0
da13p3 ONLINE 0 0 0
da12p3 ONLINE 0 0 0
da9p3 ONLINE 0 0 0
da14p3 ONLINE 0 0 0
da11p3 ONLINE 0 0 0
da5p3 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
da6p1 ONLINE 0 0 0
replacing-1 ONLINE 0 0 0
da18p1 ONLINE 0 0 0
da7p1 ONLINE 0 0 0
da16p1 ONLINE 0 0 0
da8p1 ONLINE 0 0 0
da0p1 ONLINE 0 0 0
da1p1 ONLINE 0 0 0
da17p1 ONLINE 0 0 0
da19p1 ONLINE 0 0 0
da2p1 ONLINE 0 0 0
da20p1 ONLINE 0 0 0
errors: No known data errors
[dan@knew:~] $
from /var/log/auth.log
Nov 1 16:37:02 knew sudo: dan : TTY=pts/0 ; PWD=/usr/home/dan ; USER=root ; COMMAND=/sbin/zpool replace system da18p1 da7p1
from /var/log/messages
Nov 1 16:37:02 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=9332658639709199239
These are related.
[dan@knew:~] $ zpool status system
pool: system
state: ONLINE
scan: resilvered 615G in 4h42m with 0 errors on Thu Nov 1 21:19:46 2018
config:
NAME STATE READ WRITE CKSUM
system ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da3p3 ONLINE 0 0 0
da10p3 ONLINE 0 0 0
da15p3 ONLINE 0 0 0
da4p3 ONLINE 0 0 0
da13p3 ONLINE 0 0 0
da12p3 ONLINE 0 0 0
da9p3 ONLINE 0 0 0
da14p3 ONLINE 0 0 0
da11p3 ONLINE 0 0 0
da5p3 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
da6p1 ONLINE 0 0 0
da7p1 ONLINE 0 0 0
da16p1 ONLINE 0 0 0
da8p1 ONLINE 0 0 0
da0p1 ONLINE 0 0 0
da1p1 ONLINE 0 0 0
da17p1 ONLINE 0 0 0
da19p1 ONLINE 0 0 0
da2p1 ONLINE 0 0 0
da20p1 ONLINE 0 0 0
errors: No known data errors
From /var/log/messages
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=5892227802261634203
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=9332658639709199239
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=250004220145174872
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=6216472763074854678
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=12795310201775582855
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=13315402097660581553
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=5840140110512920130
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=13603535286907309607
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=4677401754715191854
Nov 1 21:19:47 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=6179550455338697142
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment