Instantly share code, notes, and snippets.

@dlangille /gist:b84993057299fa130f74c1c26f75b016
Last active Nov 6, 2018

Embed
What would you like to do?
Download ZIP
Raw
gistfile1.txt
Nov 5 15:29:33 knew getty[30735]: open /dev/ttyu2: No such file or directory
Nov 5 15:58:11 knew smartd[1068]: Device: /dev/da16 [SAT], FAILED SMART self-check. BACK UP DATA NOW!
Nov 5 15:58:11 knew smartd[1068]: Device: /dev/da16 [SAT], Failed SMART usage Attribute: 240 Head_Flying_Hours.
OK, let's replace da20 (now on the system as da16)
sudo zpool replace system gpt/653BK12FFS9A.r1.c3 gpt/57NGK1ZGF57D.r1.c3
[dan@knew:~] $ zpool status system
pool: system
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Nov 5 16:00:57 2018
221G scanned out of 45.2T at 353M/s, 37h8m to go
13.0G resilvered, 0.48% done
config:
NAME STATE READ WRITE CKSUM
system ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gpt/X643KHBFF57D.r5.c4 ONLINE 0 0 0
gpt/4728K24SF57D.r3.c2 ONLINE 0 0 0
gpt/37KVK1JRF57D.r2.c1 ONLINE 0 0 0
gpt/37D4KBJPF57D.r5.c3 ONLINE 0 0 0
gpt/5782KL6VF57D.r2.c2 ONLINE 0 0 0
gpt/6525K2DGFS9A.r2.c4 ONLINE 0 0 0
gpt/579HKDZYF57D.r3.c3 ONLINE 0 0 0
gpt/579HKDZXF57D.r2.c3 ONLINE 0 0 0
gpt/5782KL6MF57D.r3.c1 ONLINE 0 0 0
gpt/X6IEKELNF57D.r4.c4 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gpt/653BK12JFS9A.r4.c2 ONLINE 0 0 0
gpt/579IK5RMF57D.r4.c3 ONLINE 0 0 0
gpt/653EK93PFS9A.r1.c4 ONLINE 0 0 0
gpt/653DK7WPFS9A.r3.c4 ONLINE 0 0 0
gpt/653DK7WCFS9A.r4.c1 ONLINE 0 0 0
gpt/653EK93QFS9A.r5.c2 ONLINE 0 0 0
gpt/653AK2MXFS9A.r1.c2 ONLINE 0 0 0
gpt/6539K3OJFS9A.r1.c1 ONLINE 0 0 0
gpt/653IK1IBFS9A.r5.c1 ONLINE 0 0 0
replacing-9 ONLINE 0 0 0
gpt/653BK12FFS9A.r1.c3 ONLINE 0 0 0
gpt/57NGK1ZGF57D.r1.c3 ONLINE 0 0 0
errors: No known data errors
Nov 5 16:00:56 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=1933292688604201684
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): WRITE(16). CDB: 8a 00 00 00 00 01 a3 93 11 30 00 00 00 08 00 00 length 4096 SMID 474 terminated ioc 804b loginfo 31110d00 scsi 0 state c xfer 0
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 07 f9 11 d0 00 00 60 00 length 49152 SMID 262 terminated ioc 804b loginfo 31110d00 (da15:mps1:0:30:0): WRITE(16). CDB: 8a 00 00 00 00 01 a3 93 11 30 00 00 00 08 00 00
Nov 5 16:10:54 knew kernel: scsi 0 state c xfer 0
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): CAM status: CCB request completed with an error
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): Retrying command
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 07 f9 11 d0 00 00 60 00
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): CAM status: CCB request completed with an error
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): Retrying command
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 07 f9 11 d0 00 00 60 00
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): CAM status: SCSI Status Error
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): SCSI status: Check Condition
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): Retrying command (per sense data)
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): WRITE(16). CDB: 8a 00 00 00 00 01 a3 93 7f e8 00 00 00 10 00 00
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): CAM status: SCSI Status Error
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): SCSI status: Check Condition
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Nov 5 16:10:54 knew kernel: (da15:mps1:0:30:0): Retrying command (per sense data)
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): WRITE(16). CDB: 8a 00 00 00 00 01 a4 17 30 10 00 00 00 08 00 00 length 4096 SMID 899 terminated ioc 804b loginfo 31110d00 scsi 0 state c xfer 0
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 41 39 c3 20 00 00 88 00 length 69632 SMID 393 terminated ioc 804b loginfo 31110d00 (da15:mps1:0:30:0): WRITE(16). CDB: 8a 00 00 00 00 01 a4 17 30 10 00 00 00 08 00 00
Nov 5 16:21:09 knew kernel: scsi 0 state c xfer 0
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): CAM status: CCB request completed with an error
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): Retrying command
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 41 39 c3 20 00 00 88 00
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): CAM status: CCB request completed with an error
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): Retrying command
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 41 39 c3 20 00 00 88 00
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): CAM status: SCSI Status Error
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): SCSI status: Check Condition
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Nov 5 16:21:09 knew kernel: (da15:mps1:0:30:0): Retrying command (per sense data)
Nov 5 16:21:10 knew kernel: (da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 41 39 f5 98 00 00 c8 00
Nov 5 16:21:10 knew kernel: (da15:mps1:0:30:0): CAM status: SCSI Status Error
Nov 5 16:21:10 knew kernel: (da15:mps1:0:30:0): SCSI status: Check Condition
Nov 5 16:21:10 knew kernel: (da15:mps1:0:30:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Nov 5 16:21:10 knew kernel: (da15:mps1:0:30:0): Retrying command (per sense data)
Nov 5 16:21:28 knew kernel: (da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 3f 93 92 08 00 01 00 00 length 131072 SMID 667 terminated ioc 804b loginfo 31110d00 scsi 0 state c xfer 0
Nov 5 16:21:28 knew kernel: (da15:mps1:0:30:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 678 terminated ioc 804b loginfo 3(da15:mps1:0:30:0): WRITE(10). CDB: 2a 00 3f 93 92 08 00 01 00 00
Nov 5 16:21:28 knew kernel: 1110d00 scsi 0 state c xfer 0
Nov 5 16:21:28 knew kernel: (da15:mps1:0:30:0): CAM status: CCB request completed with an error
Nov 5 16:21:28 knew kernel: (da15:mps1:0:30:0): Retrying command
Nov 5 16:21:28 knew kernel: (da15:mps1:0:30:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
Nov 5 16:21:28 knew kernel: (da15:mps1:0:30:0): CAM status: CCB request completed with an error
Nov 5 16:21:28 knew kernel: (da15:mps1:0:30:0): Retrying command
Nov 5 16:21:29 knew kernel: (da15:mps1:0:30:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
Nov 5 16:21:29 knew kernel: (da15:mps1:0:30:0): CAM status: SCSI Status Error
Nov 5 16:21:29 knew kernel: (da15:mps1:0:30:0): SCSI status: Check Condition
Nov 5 16:21:29 knew kernel: (da15:mps1:0:30:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Nov 5 16:21:29 knew kernel: (da15:mps1:0:30:0): Error 6, Retries exhausted
Nov 5 16:21:29 knew kernel: (da15:mps1:0:30:0): Invalidating pack
Nov 5 16:21:29 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=1933292688604201684
Nov 5 16:21:29 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=1933292688604201684
bash: [dan@knew:~]: command not found
[dan@knew:~] $ zpool status system
pool: system
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Nov 5 16:00:57 2018
560G scanned out of 45.2T at 360M/s, 36h9m to go
26.0G resilvered, 1.21% done
config:
NAME STATE READ WRITE CKSUM
system DEGRADED 0 0 0
raidz2-0 ONLINE 0 0 0
gpt/X643KHBFF57D.r5.c4 ONLINE 0 0 0
gpt/4728K24SF57D.r3.c2 ONLINE 0 0 0
gpt/37KVK1JRF57D.r2.c1 ONLINE 0 0 0
gpt/37D4KBJPF57D.r5.c3 ONLINE 0 0 0
gpt/5782KL6VF57D.r2.c2 ONLINE 0 0 0
gpt/6525K2DGFS9A.r2.c4 ONLINE 0 0 0
gpt/579HKDZYF57D.r3.c3 ONLINE 0 0 0
gpt/579HKDZXF57D.r2.c3 ONLINE 0 0 0
gpt/5782KL6MF57D.r3.c1 ONLINE 0 0 0
gpt/X6IEKELNF57D.r4.c4 ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
gpt/653BK12JFS9A.r4.c2 ONLINE 0 0 0
gpt/579IK5RMF57D.r4.c3 ONLINE 0 0 0
gpt/653EK93PFS9A.r1.c4 ONLINE 0 0 0
gpt/653DK7WPFS9A.r3.c4 ONLINE 0 0 0
gpt/653DK7WCFS9A.r4.c1 ONLINE 0 0 0
gpt/653EK93QFS9A.r5.c2 ONLINE 0 0 0
gpt/653AK2MXFS9A.r1.c2 ONLINE 0 0 0
gpt/6539K3OJFS9A.r1.c1 ONLINE 0 0 0
gpt/653IK1IBFS9A.r5.c1 ONLINE 0 0 0
replacing-9 DEGRADED 0 0 0
gpt/653BK12FFS9A.r1.c3 ONLINE 0 0 0
gpt/57NGK1ZGF57D.r1.c3 FAULTED 6 111 0 too many errors
errors: No known data errors
[dan@knew:~] $
@dlangille

This comment has been minimized.

Sign in to view
Owner

dlangille commented Nov 6, 2018

All good now.

[dan@knew:~] $ zpool status system
  pool: system
 state: ONLINE
  scan: resilvered 630G in 6h59m with 0 errors on Mon Nov  5 23:00:14 2018
config:

	NAME                          STATE     READ WRITE CKSUM
	system                        ONLINE       0     0     0
	  raidz2-0                    ONLINE       0     0     0
	    gpt/X643KHBFF57D.r5.c4    ONLINE       0     0     0
	    gpt/4728K24SF57D.r3.c2    ONLINE       0     0     0
	    gpt/37KVK1JRF57D.r2.c1    ONLINE       0     0     0
	    gpt/37D4KBJPF57D.r5.c3    ONLINE       0     0     0
	    gpt/5782KL6VF57D.r2.c2    ONLINE       0     0     0
	    gpt/6525K2DGFS9A.r2.c4    ONLINE       0     0     0
	    gpt/579HKDZYF57D.r3.c3    ONLINE       0     0     0
	    gpt/579HKDZXF57D.r2.c3    ONLINE       0     0     0
	    gpt/5782KL6MF57D.r3.c1    ONLINE       0     0     0
	    gpt/X6IEKELNF57D.r4.c4    ONLINE       0     0     0
	  raidz2-1                    ONLINE       0     0     0
	    gpt/653BK12JFS9A.r4.c2    ONLINE       0     0     0
	    gpt/579IK5RMF57D.r4.c3    ONLINE       0     0     0
	    gpt/653EK93PFS9A.r1.c4    ONLINE       0     0     0
	    gpt/653DK7WPFS9A.r3.c4    ONLINE       0     0     0
	    gpt/653DK7WCFS9A.r4.c1    ONLINE       0     0     0
	    gpt/653EK93QFS9A.r5.c2    ONLINE       0     0     0
	    gpt/653AK2MXFS9A.r1.c2    ONLINE       0     0     0
	    gpt/6539K3OJFS9A.r1.c1    ONLINE       0     0     0
	    gpt/653IK1IBFS9A.r5.c1    ONLINE       0     0     0
	    replacing-9               ONLINE       0     0     2
	      da15p1                  ONLINE       0     0     0
	      gpt/57NGK1ZGF57D.r1.c3  ONLINE       0     0     0

errors: No known data errors
[dan@knew:~] $

But why:

  • the name changed after the reboot
  • the original device wasn't removed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment