Skip to content

Instantly share code, notes, and snippets.

@dlangille

dlangille/01 - existing 3TB drive

Last active July 25, 2017 11:08
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dlangille/15f96440232415598fa6b20e88b9c071 to your computer and use it in GitHub Desktop.
Save dlangille/15f96440232415598fa6b20e88b9c071 to your computer and use it in GitHub Desktop.
Download ZIP
Swapping 3TB out, swapping 5TB in
Raw
01 - existing 3TB drive
$ gpart show da2
=> 34 5860533101 da2 GPT (2.7T)
34 94 1 freebsd-boot (47K)
128 16777216 2 freebsd-swap (8.0G)
16777344 5838471168 3 freebsd-zfs (2.7T)
5855248512 5284623 - free - (2.5G)
Raw
02 - For the new drives
freebsd-boot of 512K
swap of 8G
freebsd-zfs of
Raw
03 - the first insert
Jul 5 21:50:12 slocum kernel: mps0: SAS Address for SATA device = d53d5a18adac7082
Jul 5 21:50:12 slocum kernel: mps0: SAS Address from SATA device = d53d5a18adac7082
Jul 5 21:50:12 slocum kernel: (da6:mps0:0:25:0): UNMAPPED
Jul 5 21:50:12 slocum kernel: da6 at mps0 bus 0 scbus0 target 25 lun 0
Jul 5 21:50:12 slocum kernel: da6: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SPC-4 SCSI device
Jul 5 21:50:12 slocum kernel: da6: Serial Number 653DK7WBFS9A
Jul 5 21:50:12 slocum kernel: da6: 600.000MB/s transfers
Jul 5 21:50:12 slocum kernel: da6: Command Queueing enabled
Jul 5 21:50:12 slocum kernel: da6: 4769307MB (9767541168 512 byte sectors)
Raw
04 - creating partitions
sudo gpart add -s 512K -t freebsd-boot -a 1M da6
sudo gpart add -s 4G -t freebsd-swap -a 1M da6
sudo gpart add -s 4653G -t freebsd-zfs -a 1M da6
$ gpart show da6
=> 34 9767541101 da6 GPT (4.5T)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 8388608 2 freebsd-swap (4.0G)
8390656 9758048256 3 freebsd-zfs (4.5T)
9766438912 1102223 - free - (538M)
Raw
05 - smartctl
$ sudo smartctl -a /dev/da6
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MD04ACA... Enterprise HDD
Device Model: TOSHIBA MD04ACA500
Serial Number: 653DK7WBFS9A
LU WWN Device Id: 5 000039 65bd0049c
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Jul 6 13:42:12 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 548) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8902
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 24
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 079 079 000 Old_age Always - 8779
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 24
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 14
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 5371
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 43 (Min/Max 19/51)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 082 082 000 Old_age Always - 7444
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 295
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 10 -
# 2 Short offline Completed without error 00% 1 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Raw
06 - HDWE150 from MacSales
[dan@slocum:~] $ sudo smartctl -a /dev/da6
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: TOSHIBA HDWE150
Serial Number: 4725K6ECF57D
LU WWN Device Id: 5 000039 7bb90031c
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Jul 7 18:39:35 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 546) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8609
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 2
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 0
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 0
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 29 (Min/Max 26/29)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 100 100 000 Old_age Always - 0
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 562
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
[dan@slocum:~] $
Raw
07 - HDWE150 from MacSales
[dan@slocum:~] $ sudo smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: TOSHIBA HDWE150
Serial Number: 4725K6DVF57D
LU WWN Device Id: 5 000039 7bb900240
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Jul 7 18:43:29 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 542) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8653
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 1
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 0
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 1
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 0
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 46 (Min/Max 28/46)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 100 100 000 Old_age Always - 0
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 557
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
[dan@slocum:~] $
Raw
08 - MD04ACA500
$ sudo smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MD04ACA... Enterprise HDD
Device Model: TOSHIBA MD04ACA500
Serial Number: 652FK58FFS9A
LU WWN Device Id: 5 000039 65be000ee
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jul 8 12:11:57 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 545) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8699
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 29
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 48
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 079 079 000 Old_age Always - 8786
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 29
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 5
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 22
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 448
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 39 (Min/Max 19/51)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 4
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 079 079 000 Old_age Always - 8683
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 213
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 9 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
[dan@slocum:~] $
Raw
09 - first drive replaced
# pulled one drive (da0)
# inserted another
$ zpool status
pool: system
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0 in 35h54m with 0 errors on Tue Jul 4 14:58:14 2017
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
gpt/disk4 ONLINE 0 0 0
4885020496131451443 REMOVED 0 0 0 was /dev/gpt/disk5
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
$ sudo zpool replace system gpt/disk5 da0p3
$ zpool status
pool: system
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Jul 8 12:19:10 2017
436M scanned out of 6.13T at 9.09M/s, 196h28m to go
58.9M resilvered, 0.01% done
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
gpt/disk4 ONLINE 0 0 0
replacing-5 REMOVED 0 0 0
4885020496131451443 REMOVED 0 0 0 was /dev/gpt/disk5
da0p3 ONLINE 0 0 0 (resilvering)
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
Raw
10 - second drive being replaced
Jul 11 14:54:17 slocum kernel: mps0: mpssas_prepare_remove: Sending reset for target ID 27
Jul 11 14:54:17 slocum kernel: da6 at mps0 bus 0 scbus0 target 27 lun 0
Jul 11 14:54:17 slocum kernel: da6: <ATA TOSHIBA HDWE150 FP2A> s/n 4725K6ECF57D detached
Jul 11 14:54:17 slocum kernel: (da6:mps0:0:27:0): Periph destroyed
Jul 11 14:54:17 slocum kernel: mps0: Unfreezing devq for target ID 27
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): READ(16). CDB: 88 00 00 00 00 01 36 fc 8d 68 00 00 00 08 00 00 length 4096 SMID 351 terminated ioc 804b scsi 0 state c xfer 0
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): READ(16). CDB: 88 00 00 00 00 01 36 fc 8d 68 00 00 00 08 00 00
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): CAM status: CCB request completed with an error
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): Retrying command
Jul 11 14:54:30 slocum kernel: mps0: mpssas_prepare_remove: Sending reset for target ID 20
Jul 11 14:54:30 slocum kernel: da1 at mps0 bus 0 scbus0 target 20 lun 0
Jul 11 14:54:30 slocum kernel: da1: <ATA Hitachi HDS72303 AAB0> s/n MSM9215Q0BEEUY detached
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): WRITE(10). CDB: 2a 00 26 b1 db 00 00 00 08 00
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): CAM status: CCB request aborted by the host
Jul 11 14:54:30 slocum kernel: mps0: (da1:Unfreezing devq for target ID 20
Jul 11 14:54:30 slocum kernel: mps0:0:20:0): Error 5, Periph was invalidated
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): READ(16). CDB: 88 00 00 00 00 01 36 fc 8d 68 00 00 00 08 00 00
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): CAM status: CCB request aborted by the host
Jul 11 14:54:30 slocum kernel: (da1:mps0:0:20:0): Error 5, Periph was invalidated
Jul 11 14:54:30 slocum kernel: GEOM_MIRROR: Device swap: provider da1p2 disconnected.
Jul 11 14:54:31 slocum kernel: (da1:mps0:0:20:0): Periph destroyed
Jul 11 14:54:49 slocum kernel: mps0: SAS Address for SATA device = 452e370496809487
Jul 11 14:54:49 slocum kernel: mps0: SAS Address from SATA device = 452e370496809487
Jul 11 14:54:49 slocum kernel: (da1:mps0:0:27:0): UNMAPPED
Jul 11 14:54:49 slocum kernel: da1 at mps0 bus 0 scbus0 target 27 lun 0
Jul 11 14:54:49 slocum kernel: da1: <ATA TOSHIBA HDWE150 FP2A> Fixed Direct Access SPC-4 SCSI device
Jul 11 14:54:49 slocum kernel: da1: Serial Number 4725K6ECF57D
Jul 11 14:54:49 slocum kernel: da1: 600.000MB/s transfers
Jul 11 14:54:49 slocum kernel: da1: Command Queueing enabled
Jul 11 14:54:49 slocum kernel: da1: 4769307MB (9767541168 512 byte sectors)
[dan@slocum:~] $ zpool status
pool: system
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0 in 22h1m with 0 errors on Tue Jul 11 01:32:55 2017
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
8194939911233312160 REMOVED 0 0 0 was /dev/gpt/disk4
da0p3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $
[dan@slocum:~] $ sudo zpool replace system gpt/disk4 da1p3
Make sure to wait until resilver is done before rebooting.
If you boot from pool 'system', you may need to update
boot code on newly attached disk 'da1p3'.
Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
[dan@slocum:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
partcode written to da1p1
bootcode written to da1
[dan@slocum:~] $ zpool status
pool: system
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Jul 11 14:56:52 2017
306M scanned out of 6.04T at 9.57M/s, 183h50m to go
37.2M resilvered, 0.00% done
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
replacing-4 REMOVED 0 0 0
8194939911233312160 REMOVED 0 0 0 was /dev/gpt/disk4
da1p3 ONLINE 0 0 0 (resilvering)
da0p3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $
Raw
11 - 4725K6DVF57D
Jul 11 15:22:36 slocum kernel: mps0: SAS Address for SATA device = 452e361796809487
Jul 11 15:22:36 slocum kernel: mps0: SAS Address from SATA device = 452e361796809487
Jul 11 15:22:36 slocum kernel: (da6:mps0:0:26:0): UNMAPPED
Jul 11 15:22:36 slocum kernel: da6 at mps0 bus 0 scbus0 target 26 lun 0
Jul 11 15:22:36 slocum kernel: da6: <ATA TOSHIBA HDWE150 FP2A> Fixed Direct Access SPC-4 SCSI device
Jul 11 15:22:36 slocum kernel: da6: Serial Number 4725K6DVF57D
Jul 11 15:22:36 slocum kernel: da6: 600.000MB/s transfers
Jul 11 15:22:36 slocum kernel: da6: Command Queueing enabled
Jul 11 15:22:36 slocum kernel: da6: 4769307MB (9767541168 512 byte sectors)
[dan@slocum:~] $ sudo smartctl -a /dev/da6
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: TOSHIBA HDWE150
Serial Number: 4725K6DVF57D
LU WWN Device Id: 5 000039 7bb900240
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Jul 11 15:50:02 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 542) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8641
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 2
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 18
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 1
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 3
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 43 (Min/Max 28/51)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 100 100 000 Old_age Always - 8
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 549
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 8 -
# 2 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
[dan@slocum:~] $
Raw
12 - 3rd drive in 4725K6DVF57D
Removing the new 5TB drives from where gpart was run:
Jul 17 20:02:08 slocum kernel: mps0: mpssas_prepare_remove: Sending reset for target ID 26
Jul 17 20:02:08 slocum kernel: da5 at mps0 bus 0 scbus0 target 26 lun 0
Jul 17 20:02:08 slocum kernel: da5: <ATA TOSHIBA HDWE150 FP2A> s/n 4725K6DVF57D detached
Jul 17 20:02:08 slocum kernel: (da5:mps0:0:26:0): Periph destroyed
Jul 17 20:02:08 slocum kernel: mps0: Unfreezing devq for target ID 26
Removing the 3TB drive which it will replace:
Jul 17 20:05:13 slocum kernel: (da0:mps0:0:21:0): WRITE(10). CDB: 2a 00 28 5c d6 30 00 00 08 00 length 4096 SMID 134 terminated ioc 804b scsi 0 state c xfer 0
Jul 17 20:05:13 slocum kernel: (da0:mps0:0:21:0): WRITE(10). CDB: 2a 00 28 5c d6 30 00 00 08 00
Jul 17 20:05:13 slocum kernel: (da0:mps0:0:21:0): CAM status: CCB request completed with an error
Jul 17 20:05:13 slocum kernel: (da0:mps0:0:21:0): Retrying command
Jul 17 20:05:15 slocum kernel: mps0: mpssas_prepare_remove: Sending reset for target ID 21
Jul 17 20:05:15 slocum kernel: da0 at mps0 bus 0 scbus0 target 21 lun 0
Jul 17 20:05:15 slocum kernel: da0: <ATA Hitachi HDS72303 AAB0> s/n MSM9215Q0BDB5Y detached
Jul 17 20:05:15 slocum kernel: (da0:mps0:0:21:0): WRITE(10). CDB: 2a 00 28 5c d6 30 00 00 08 00
Jul 17 20:05:15 slocum kernel: mps0: (da0:mps0:0:21:0): CAM status: CCB request aborted by the host
Jul 17 20:05:15 slocum kernel: Unfreezing devq for target ID 21
Jul 17 20:05:15 slocum kernel: (da0:mps0:0:21:0): Error 5, Periph was invalidated
Jul 17 20:05:15 slocum kernel: GEOM_MIRROR: Device swap: provider da0p2 disconnected.
Jul 17 20:05:15 slocum kernel: (da0:mps0:0:21:0): Periph destroyed
Jul 17 20:05:15 slocum ZFS: vdev is removed, pool_guid=15387115135938424988 vdev_guid=3597532040953426928
Adding the 5TB drive into the above slot:
Jul 17 20:05:39 slocum kernel: mps0: SAS Address for SATA device = 452e361796809487
Jul 17 20:05:39 slocum kernel: mps0: SAS Address from SATA device = 452e361796809487
Jul 17 20:05:39 slocum kernel: (da0:mps0:0:26:0): UNMAPPED
Jul 17 20:05:39 slocum kernel: da0 at mps0 bus 0 scbus0 target 26 lun 0
Jul 17 20:05:39 slocum kernel: da0: <ATA TOSHIBA HDWE150 FP2A> Fixed Direct Access SPC-4 SCSI device
Jul 17 20:05:39 slocum kernel: da0: Serial Number 4725K6DVF57D
Jul 17 20:05:39 slocum kernel: da0: 600.000MB/s transfers
Jul 17 20:05:39 slocum kernel: da0: Command Queueing enabled
Jul 17 20:05:39 slocum kernel: da0: 4769307MB (9767541168 512 byte sectors)
[dan@slocum:~] $ zpool status
pool: system
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 820G in 23h32m with 0 errors on Wed Jul 12 14:29:45 2017
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gpt/disk0 ONLINE 0 0 0
3597532040953426928 REMOVED 0 0 0 was /dev/gpt/disk1
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
replacing in ZFS
[dan@slocum:~] $ sudo zpool replace system gpt/disk1 da0p3
Make sure to wait until resilver is done before rebooting.
If you boot from pool 'system', you may need to update
boot code on newly attached disk 'da0p3'.
Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
writing the bootcode:
[dan@slocum:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
partcode written to da0p1
bootcode written to da0
[dan@slocum:~] $
pool: system
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Jul 17 20:08:17 2017
552M scanned out of 6.08T at 9.51M/s, 186h14m to go
68.5M resilvered, 0.01% done
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gpt/disk0 ONLINE 0 0 0
replacing-1 REMOVED 0 0 0
3597532040953426928 REMOVED 0 0 0 was /dev/gpt/disk1
da0p3 ONLINE 0 0 0 (resilvering)
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
Raw
12 - gstat during 2nd drive replace
dT: 1.001s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 0 0 0 0.0 0 0 0.0 0.0| ada0
0 0 0 0 0.0 0 0 0.0 0.0| ada1
1 1458 1458 25794 0.5 0 0 0.0 24.4| da2
2 1249 1249 26441 0.2 0 0 0.0 13.9| da3
1 3505 3505 64476 0.8 0 0 0.0 72.8| da4
0 2510 2510 38798 0.4 0 0 0.0 39.4| da5
2 2981 2981 63809 0.6 0 0 0.0 60.8| da0
0 1082 0 0 0.0 1082 35605 0.3 36.7| da1
0 0 0 0 0.0 0 0 0.0 0.0| da6
0 0 0 0 0.0 0 0 0.0 0.0| da7
Raw
13 - 4725K6ECF57D long test completed
[dan@slocum:~] $ sudo smartctl -a /dev/da6
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: TOSHIBA HDWE150
Serial Number: 4725K6ECF57D
LU WWN Device Id: 5 000039 7bb90031c
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jul 10 15:43:05 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 546) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8629
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 3
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 68
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 3
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 1
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 4
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 46 (Min/Max 26/53)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 100 100 000 Old_age Always - 16
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 574
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 25 -
# 2 Extended offline Completed without error 00% 8 -
# 3 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Raw
14 - 652FK58FFS9A long test comp
[dan@slocum:~] $ sudo smartsudo smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MD04ACA... Enterprise HDD
Device Model: TOSHIBA MD04ACA500
Serial Number: 652FK58FFS9A
LU WWN Device Id: 5 000039 65be000ee
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jul 10 15:42:51 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 545) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8699
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 29
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 48
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 078 078 000 Old_age Always - 8838
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 29
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 5
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 22
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 450
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 50 (Min/Max 19/54)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 4
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 079 079 000 Old_age Always - 8728
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 298
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 8795 -
# 2 Short offline Completed without error 00% 8786 -
# 3 Extended offline Completed without error 00% 9 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
[dan@slocum:~] $
Raw
15 - resilver done
[dan@slocum:~] $ zpool status
pool: system
state: ONLINE
scan: resilvered 822G in 21h27m with 0 errors on Tue Jul 18 17:35:47 2017
config:
NAME STATE READ WRITE CKSUM
system ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gpt/disk0 ONLINE 0 0 0
da0p3 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $
Raw
16 - current gpart
[dan@slocum:~] $ gpart show
=> 34 62533229 ada0 GPT (30G)
34 6 - free - (3.0K)
40 16777216 1 freebsd-zfs (8.0G)
16777256 45756007 - free - (22G)
=> 34 62533229 ada1 GPT (30G)
34 6 - free - (3.0K)
40 16777216 1 freebsd-zfs (8.0G)
16777256 45756007 - free - (22G)
=> 34 5860533101 da1 GPT (2.7T)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 4193368 2 freebsd-swap (2.0G)
4194432 5851054080 3 freebsd-zfs (2.7T)
5855248512 5284623 - free - (2.5G)
=> 34 5860533101 da2 GPT (2.7T)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 4193368 2 freebsd-swap (2.0G)
4194432 5851054080 3 freebsd-zfs (2.7T)
5855248512 5284623 - free - (2.5G)
=> 34 5860533101 da3 GPT (2.7T)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 4193368 2 freebsd-swap (2.0G)
4194432 5851054080 3 freebsd-zfs (2.7T)
5855248512 5284623 - free - (2.5G)
=> 34 9767541101 diskid/DISK-653DK7WBFS9A GPT (4.5T)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 8388608 2 freebsd-swap (4.0G)
8390656 9758048256 3 freebsd-zfs (4.5T)
9766438912 1102223 - free - (538M)
=> 40 9767541088 diskid/DISK-4725K6ECF57D GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
=> 40 9767541088 da0 GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
=> 40 9767541088 da7 GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
=> 40 9767541088 diskid/DISK-652FK58FFS9A GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
=> 40 9767541088 da5 GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
=> 40 9767541088 diskid/DISK-653BK12IFS9A GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
[dan@slocum:~] $
Raw
17 - adding in 652FK58FFS9A
[dan@slocum:~] $ gpart show da7
=> 40 9767541088 da7 GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
[dan@slocum:~] $
[dan@slocum:~] $ sudo smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MD04ACA... Enterprise HDD
Device Model: TOSHIBA MD04ACA500
Serial Number: 652FK58FFS9A
LU WWN Device Id: 5 000039 65be000ee
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Jul 20 13:29:01 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 545) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 2817
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 33
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 48
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 078 078 000 Old_age Always - 9075
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 33
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 5
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 26
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 761
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 48 (Min/Max 19/54)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 4
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 078 078 000 Old_age Always - 8846
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 204
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 8795 -
# 2 Short offline Completed without error 00% 8786 -
# 3 Extended offline Completed without error 00% 9 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
[dan@slocum:~] $
after removing drive from zpool and inserting new drive:
[dan@slocum:~] $ zpool status
pool: system
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0 in 22h10m with 0 errors on Thu Jul 20 01:14:15 2017
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
3558867368789024889 REMOVED 0 0 0 was /dev/gpt/disk0
da0p3 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $
new drive being inserted:
Jul 20 13:31:13 slocum kernel: mps0: SAS Address for SATA device = d537351fadac7080
Jul 20 13:31:13 slocum kernel: mps0: SAS Address from SATA device = d537351fadac7080
Jul 20 13:31:14 slocum kernel: (da5:mps0:0:29:0): UNMAPPED
Jul 20 13:31:14 slocum kernel: da5 at mps0 bus 0 scbus0 target 29 lun 0
Jul 20 13:31:14 slocum kernel: da5: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SPC-4 SCSI device
Jul 20 13:31:14 slocum kernel: da5: Serial Number 653BK12IFS9A
Jul 20 13:31:14 slocum kernel: da5: 600.000MB/s transfers
Jul 20 13:31:14 slocum kernel: da5: Command Queueing enabled
Jul 20 13:31:14 slocum kernel: da5: 4769307MB (9767541168 512 byte sectors)
[dan@slocum:~] $ gpart show da5
=> 40 9767541088 da5 GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
[dan@slocum:~] $ sudo zpool replace system gpt/disk0 da5p3
Make sure to wait until resilver is done before rebooting.
If you boot from pool 'system', you may need to update
boot code on newly attached disk 'da5p3'.
Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
[dan@slocum:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da5
partcode written to da5p1
bootcode written to da5
[dan@slocum:~] $
[dan@slocum:~] $ zpool status
pool: system
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Jul 20 13:40:09 2017
614M scanned out of 6.08T at 9.30M/s, 190h32m to go
78.8M resilvered, 0.01% done
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
replacing-0 REMOVED 0 0 0
3558867368789024889 REMOVED 0 0 0 was /dev/gpt/disk0
da5p3 ONLINE 0 0 0 (resilvering)
da0p3 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $
Raw
18 - resilver completed
[dan@slocum:~] $ zpool status
pool: system
state: ONLINE
scan: resilvered 816G in 23h21m with 0 errors on Fri Jul 21 13:01:42 2017
config:
NAME STATE READ WRITE CKSUM
system ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da5p3 ONLINE 0 0 0
da0p3 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
Raw
19 - 652FK58FFS9A goes in
identify the drive by pulling it out and replacing it:
Jul 21 13:08:24 slocum kernel: mps0: mpssas_prepare_remove: Sending reset for target ID 28
Jul 21 13:08:24 slocum kernel: da1 at mps0 bus 0 scbus0 target 28 lun 0
Jul 21 13:08:24 slocum kernel: da1: <ATA TOSHIBA MD04ACA5 FP2A> s/n 652FK58FFS9A detached
Jul 21 13:08:24 slocum kernel: (da1:mps0:0:28:0): Periph destroyed
Jul 21 13:08:24 slocum kernel: mps0: Unfreezing devq for target ID 28
Jul 21 13:08:27 slocum kernel: mps0: SAS Address for SATA device = d53b3b1cadac6f84
Jul 21 13:08:27 slocum kernel: mps0: SAS Address from SATA device = d53b3b1cadac6f84
Jul 21 13:08:27 slocum kernel: (da1:mps0:0:28:0): UNMAPPED
Jul 21 13:08:27 slocum kernel: da1 at mps0 bus 0 scbus0 target 28 lun 0
Jul 21 13:08:27 slocum kernel: da1: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SPC-4 SCSI device
Jul 21 13:08:27 slocum kernel: da1: Serial Number 652FK58FFS9A
Jul 21 13:08:27 slocum kernel: da1: 600.000MB/s transfers
Jul 21 13:08:27 slocum kernel: da1: Command Queueing enabled
Jul 21 13:08:27 slocum kernel: da1: 4769307MB (9767541168 512 byte sectors)
[dan@slocum:~] $ sudo smartctl -a /dev/da1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MD04ACA... Enterprise HDD
Device Model: TOSHIBA MD04ACA500
Serial Number: 652FK58FFS9A
LU WWN Device Id: 5 000039 65be000ee
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Jul 21 13:09:12 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 545) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 1850
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 35
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 48
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 078 078 000 Old_age Always - 9098
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 5
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 28
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 763
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 51 (Min/Max 19/54)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 4
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 078 078 000 Old_age Always - 8869
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 202
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 8795 -
# 2 Short offline Completed without error 00% 8786 -
# 3 Extended offline Completed without error 00% 9 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
[dan@slocum:~] $ gpart show da1
=> 40 9767541088 da1 GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
Now let's pull the drive which the above will replace:
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): READ(10). CDB: 28 00 08 40 6a 68 00 00 08 00 length 4096 SMID 635 terminated ioc 804b scsi 0 state c xfer 0
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): WRITE(10). CDB: 2a 00 28 7e aa f8 00 00 08 00 length 4096 SMID 548 terminated ioc 804b scsi 0 state c xfer(da2:mps0:0:23:0): READ(10). CDB: 28 00 08 40 6a 68 00 00 08 00
Jul 21 13:11:34 slocum kernel: 0
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): CAM status: CCB request completed with an error
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): Retrying command
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): WRITE(10). CDB: 2a 00 28 7e aa f8 00 00 08 00
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): CAM status: CCB request completed with an error
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): Retrying command
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): WRITE(10). CDB: 2a 00 28 7e aa f8 00 00 08 00 length 4096 SMID 570 terminated ioc 804b scsi 0 state 0 xfer 0
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): READ(10). CDB: 28 00 08 40 6a 68 00 00 08 00 length 4096 SMID 602 terminated ioc 804b scsi 0 state 0 xfer 0
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): WRITE(10). CDB: 2a 00 28 7e aa f8 00 00 08 00
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): CAM status: CCB request completed with an error
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): Retrying command
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): READ(10). CDB: 28 00 08 40 6a 68 00 00 08 00
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): CAM status: CCB request completed with an error
Jul 21 13:11:34 slocum kernel: (da2:mps0:0:23:0): Retrying command
Jul 21 13:11:35 slocum kernel: mps0: mpssas_prepare_remove: Sending reset for target ID 23
Jul 21 13:11:35 slocum kernel: da2 at mps0 bus 0 scbus0 target 23 lun 0
Jul 21 13:11:35 slocum kernel: da2: <ATA Hitachi HDS72303 AAB0> s/n MSM9215Q0B2WVX detached
Jul 21 13:11:35 slocum kernel: (da2:mps0:0:23:0): READ(10). CDB: 28 00 08 40 6a 68 00 00 08 00
Jul 21 13:11:35 slocum kernel: (da2:mps0:0:23:0): CAM status: CCB request aborted by the host
Jul 21 13:11:35 slocum kernel: (da2:mps0:0:23:0): Error 5, Periph was invalidated
Jul 21 13:11:35 slocum kernel: mps0: Unfreezing devq for target ID 23
Jul 21 13:11:35 slocum kernel: (da2:mps0:0:23:0): WRITE(10). CDB: 2a 00 28 7e aa f8 00 00 08 00
Jul 21 13:11:35 slocum kernel: (da2:mps0:0:23:0): CAM status: CCB request aborted by the host
Jul 21 13:11:35 slocum kernel: (da2:mps0:0:23:0): Error 5, Periph was invalidated
Jul 21 13:11:35 slocum kernel: GEOM_MIRROR: Device swap: provider da2p2 disconnected.
Jul 21 13:11:36 slocum kernel: (da2:mps0:0:23:0): Periph destroyed
Jul 21 13:11:35 slocum ZFS: vdev is removed, pool_guid=15387115135938424988 vdev_guid=8095897341669412185
[dan@slocum:~] $ zpool status
pool: system
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 816G in 23h21m with 0 errors on Fri Jul 21 13:01:42 2017
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
da5p3 ONLINE 0 0 0
da0p3 ONLINE 0 0 0
8095897341669412185 REMOVED 0 0 0 was /dev/gpt/disk2
gpt/disk3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $
[dan@slocum:~] $ sudo zpool replace system gpt/disk2 da1p3
Make sure to wait until resilver is done before rebooting.
If you boot from pool 'system', you may need to update
boot code on newly attached disk 'da1p3'.
Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
[dan@slocum:~] $ gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
gpart: /dev/da1p1: Permission denied
[dan@slocum:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
partcode written to da1p1
bootcode written to da1
[dan@slocum:~] $ zpool status
pool: system
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jul 21 13:13:03 2017
245M scanned out of 6.04T at 9.42M/s, 186h41m to go
33.5M resilvered, 0.00% done
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
da5p3 ONLINE 0 0 0
da0p3 ONLINE 0 0 0
replacing-2 REMOVED 0 0 0
8095897341669412185 REMOVED 0 0 0 was /dev/gpt/disk2
da1p3 ONLINE 0 0 0 (resilvering)
gpt/disk3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $
Raw
20 - 653EK93XFS9A goes in
[dan@slocum:~] $ sudo smartctl -a /dev/da2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MD04ACA... Enterprise HDD
Device Model: TOSHIBA MD04ACA500
Serial Number: 653EK93XFS9A
LU WWN Device Id: 5 000039 65bd80461
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jul 24 15:21:26 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 542) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 7725
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 30
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 078 078 000 Old_age Always - 8883
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 30
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 1
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 23
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 565
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 46 (Min/Max 18/52)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 079 079 000 Old_age Always - 8754
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 210
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 8847 -
# 2 Short offline Completed without error 00% 8839 -
# 3 Extended offline Completed without error 00% 9 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
=> 40 9767541088 da2 GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
=> 40 9767541088 da2 GPT (4.5T)
40 2008 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 8388608 2 freebsd-swap (4.0G)
8392704 9758048256 3 freebsd-zfs (4.5T)
9766440960 1100168 - free - (537M)
[dan@slocum:~] $ sudo zpool replace system gpt/disk3 da2p3
Make sure to wait until resilver is done before rebooting.
If you boot from pool 'system', you may need to update
boot code on newly attached disk 'da2p3'.
Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
[dan@slocum:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2
partcode written to da2p1
bootcode written to da2
[dan@slocum:~] $ zpool status
pool: system
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Jul 24 15:43:40 2017
407M scanned out of 6.05T at 10.4M/s, 169h3m to go
48.3M resilvered, 0.01% done
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
da5p3 ONLINE 0 0 0
da0p3 ONLINE 0 0 0
da1p3 ONLINE 0 0 0
replacing-3 REMOVED 0 0 0
15391662935041273970 REMOVED 0 0 0 was /dev/gpt/disk3
da2p3 ONLINE 0 0 0 (resilvering)
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $
[dan@slocum:~] $ zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
system 16.2T 6.05T 10.2T - 37% 37% 1.00x DEGRADED -
[dan@slocum:~] $
Raw
21 - after the resilver
[dan@slocum:~] $ zpool status
pool: system
state: ONLINE
scan: resilvered 813G in 14h11m with 0 errors on Tue Jul 25 05:55:34 2017
config:
NAME STATE READ WRITE CKSUM
system ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da5p3 ONLINE 0 0 0
da0p3 ONLINE 0 0 0
da1p3 ONLINE 0 0 0
da2p3 ONLINE 0 0 0
diskid/DISK-4725K6ECF57Dp3 ONLINE 0 0 0
diskid/DISK-653DK7WBFS9Ap3 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/disk_134106400102 ONLINE 0 0 0
gpt/disk_133887400683 ONLINE 0 0 0
errors: No known data errors
[dan@slocum:~] $ zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
system 27.2T 6.02T 21.2T - 22% 22% 1.00x ONLINE -
[dan@slocum:~] $
NOTE, yesterday, we have 10.2T free. Now we have 21.2T free.
@allanjude
Copy link

allanjude commented Jul 6, 2017 •
edited by dlangille

Expected something more like this:

=>        34  7814037088  ada3  GPT  (3.6T)
          34         6               - free - (3.0K)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048    12582912     2  freebsd-swap  (6.0G)
    12584960  7801450496     3  freebsd-zfs  (3.6T)
  7814035456        1672        - free -  (836K)

@dlangille
Copy link
Author

Allan: yes, that is similar to what I have now. I did not redo the gpart output above after changing my commands. It is up to date now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment