Skip to content

Instantly share code, notes, and snippets.

@jlevon
Created February 1, 2019 09:32
Show Gist options
  • Save jlevon/6cc1d71d4d551ff981f3ab11cc386187 to your computer and use it in GitHub Desktop.
Save jlevon/6cc1d71d4d551ff981f3ab11cc386187 to your computer and use it in GitHub Desktop.
Testing done for FMA-related changes
Tested by Rob Johnston as follows:
- verified fmd was up
- ran "fmadm config" and fmstat, which exercises the RPC code path in fmd
- manually triggered the usage messages for smtp-notify and snmp-notify
- enabled smtp-notify service
- configured notifications for SMF services going into maintenance
- forced the intr service to fail
- ran fmadm faulty to verify a faulty was diagnoses
- verified root got an email notification
- fixed and cleared the intrd service
- ran fmadm faulty again to verify the fault was auto-repaired
Even that was probably overkill given that these changes are pretty
trivial - should be more than enough to convince an RTI advocate that
things still work.
Captured output is below.
rob
root@openindiana:~# beadm list
BE Active Mountpoint Space Policy Created
openindiana - - 8.99M static 2019-01-31 04:58
oi-20190131 - - 41.7M static 2019-01-31 06:14
smatch NR / 9.52G static 2019-01-31 08:08
root@openindiana:~# svcs fmd
STATE STIME FMRI
online 8:11:56 svc:/system/fmd:default
root@openindiana:~# fmadm config
MODULE VERSION STATUS DESCRIPTION
cpumem-retire 1.1 active CPU/Memory Retire Agent
disk-lights 1.0 active Disk Lights Agent
disk-transport 1.1 active Disk Transport Agent
eft 1.16 active eft diagnosis engine
ext-event-transport 0.2 active External FM event transport
fabric-xlate 1.0 active Fabric Ereport Translater
fmd-self-diagnosis 1.0 active Fault Manager Self-Diagnosis
io-retire 2.0 active I/O Retire Agent
sensor-transport 1.1 active Sensor Transport Agent
ses-log-transport 1.0 active SES Log Transport Agent
software-diagnosis 0.1 active Software Diagnosis engine
software-response 0.1 active Software Response Agent
sysevent-transport 1.0 active SysEvent Transport Agent
syslog-msgs 1.1 active Syslog Messaging Agent
zfs-diagnosis 1.0 active ZFS Diagnosis Engine
zfs-retire 1.0 active ZFS Retire Agent
root@openindiana:~# /usr/lib/fm/notify/smtp-notify -x
/usr/lib/fm/notify/smtp-notify: illegal option -- x
Usage: /usr/lib/fm/notify/smtp-notify [-df] [-R <altroot>]
-d enable debug mode
-f stay in foreground
-R specify alternate root
root@openindiana:~# /usr/lib/fm/notify/snmp-notify -x
/usr/lib/fm/notify/snmp-notify: illegal option -- x
Usage: /usr/lib/fm/notify/snmp-notify [-df] [-R <altroot>]
-d enable debug mode
-f stay in foreground
-R specify alternate root
root@openindiana:~# svcadm enable smtp-notify
root@openindiana:~# svccfg -s svc:/system/svc/global:default setnotify -g \
> from-online,to-maintenance mailto:root@localhost
root@openindiana:~# svccfg listnotify -g
Event: to-maintenance (source: svc:/system/svc/global:default)
Notification Type: smtp
Active: true
to: root@localhost
Event: from-online (source: svc:/system/svc/global:default)
Notification Type: smtp
Active: true
to: root@localhost
root@openindiana:~# mv /usr/lib/intrd /usr/lib/intrd.disabled
root@openindiana:~# svcadm restart intrd
root@openindiana:~# fmstat -m ext-event-transport
NAME VALUE DESCRIPTION
false
outbound_cat1class_ireport 2 events received matching ireport.*
outbound_cat1class_list 2 events received matching list.*
outbound_cat1class_other 0 events of other classes
outbound_fwd_failure 0 events we failed to forward
outbound_fwd_success 4 events forwarded successfully
outbound_recv_calls 4 total events received for forwarding
pp_bad_ruleset 0 post-process bad ruleset
pp_badclass 0 post-processing produced invalid event class
pp_badreturn 0 inconsistent number of events returned
pp_explicitdrop 0 ruleset drops event with NULL func
pp_fanoutmax 0 post-processing produced too many events
pp_intldrop 0 post-processing requested event drop
pp_nvlallocfail 0 fmd_nvl_alloc failed
pp_nvlbuildfail 0 nvlist_add_foo failed in building event
raw_callbacks 2 total raw event callbacks from producers
raw_noattrlist 0 missing attribute list
raw_nodetector 0 unable to add detector
xprt_posted 0 protocol events posted with fmd_xprt_post
root@openindiana:~# fmadm faulty
--------------- ------------------------------------ -------------- ---------
TIME EVENT-ID MSG-ID SEVERITY
--------------- ------------------------------------ -------------- ---------
Jan 31 08:39:13 099dabf8-b699-c4fe-9711-9bec4779b86e SMF-8000-YX major
Host : openindiana
Platform : VMware-Virtual-Platform Chassis_id :
VMware-56-4d-6a-a5-3c-84-d2-b7-da-66-6c-c2-02-8e-9a-fc
Product_sn :
Fault class : defect.sunos.smf.svc.maintenance
Affects : svc:///system/intrd:default
faulted and taken out of service
Problem in : svc:///system/intrd:default
faulted and taken out of service
Description : A service failed - a start, stop or refresh method failed.
Refer to http://illumos.org/msg/SMF-8000-YX for more information.
Response : The service has been placed into the maintenance state.
Impact : svc:/system/intrd:default is unavailable.
Action : Run 'svcs -xv svc:/system/intrd:default' to determine the generic
reason why the service failed, the location of any logfiles, and
a list of other services impacted.
root@openindiana:~# cat /var/mail/root
From noaccess@openindiana.local Thu Jan 31 08:39:13 2019
Return-Path: <noaccess@openindiana.local>
Received: from openindiana.local (openindiana [127.0.0.1])
by openindiana.local (8.15.2+Sun/8.15.2) with ESMTP id x0VGdDvT101979
for <root@openindiana.local>; Thu, 31 Jan 2019 08:39:13 -0800 (PST)
Received: (from noaccess@localhost)
by openindiana.local (8.15.2+Sun/8.15.2/Submit) id x0VGdDpH101977
for root@localhost; Thu, 31 Jan 2019 08:39:13 -0800 (PST)
Date: Thu, 31 Jan 2019 08:39:13 -0800 (PST)
From: No Access User <noaccess@openindiana.local>
Message-Id: <201901311639.x0VGdDpH101977@openindiana.local>
X-FMEV-HOSTNAME: openindiana
X-FMEV-CLASS: ireport.os.smf.state-transition.maintenance
X-FMEV-CODE: SMF-8000-X2
X-FMEV-SEVERITY: major
X-FMEV-FMRI: svc:/system/intrd:default
X-FMEV-FROM-STATE: offline
X-FMEV-TO-STATE: maintenance
Reply-To: root@openindiana.local
Subject: openindiana: svc:/system/intrd:default offline->maintenance
Content-Length: 332
HOSTNAME: openindiana
TIMESTAMP: January 31, 2019 at 08:39:13 AM
FMRI: svc:/system/intrd:default
FROM-STATE: offline
TO-STATE: maintenance
DESCRIPTION: The indicated service has transitioned to the maintenance state
Refer to http://illumos.org/msg/SMF-8000-X2 for more information.
REASON: a start, stop or refresh method failed
From noaccess@openindiana.local Thu Jan 31 08:39:13 2019
Return-Path: <noaccess@openindiana.local>
Received: from openindiana.local (openindiana [127.0.0.1])
by openindiana.local (8.15.2+Sun/8.15.2) with ESMTP id x0VGdDEo101980
for <root@openindiana.local>; Thu, 31 Jan 2019 08:39:13 -0800 (PST)
Received: (from noaccess@localhost)
by openindiana.local (8.15.2+Sun/8.15.2/Submit) id x0VGdDir101978
for root@localhost; Thu, 31 Jan 2019 08:39:13 -0800 (PST)
Date: Thu, 31 Jan 2019 08:39:13 -0800 (PST)
From: No Access User <noaccess@openindiana.local>
Message-Id: <201901311639.x0VGdDir101978@openindiana.local>
X-FMEV-HOSTNAME: openindiana
X-FMEV-CLASS: list.suspect
X-FMEV-UUID: 099dabf8-b699-c4fe-9711-9bec4779b86e
X-FMEV-CODE: SMF-8000-YX
X-FMEV-SEVERITY: major
Reply-To: root@openindiana.local
Subject: Fault Management Event: openindiana:SMF-8000-YX
Content-Length: 740
SUNW-MSG-ID: SMF-8000-YX, TYPE: defect, VER: 1, SEVERITY: major
EVENT-TIME: Thu Jan 31 08:39:13 PST 2019
PLATFORM: VMware-Virtual-Platform, CSN:
VMware-56-4d-6a-a5-3c-84-d2-b7-da-66-6c-c2-02-8e-9a-fc, HOSTNAME:
openindiana
SOURCE: software-diagnosis, REV: 0.1
EVENT-ID: 099dabf8-b699-c4fe-9711-9bec4779b86e
DESC: A service failed - a start, stop or refresh method failed.
Refer to http://illumos.org/msg/SMF-8000-YX for more information.
AUTO-RESPONSE: The service has been placed into the maintenance state.
IMPACT: svc:/system/intrd:default is unavailable.
REC-ACTION: Run 'svcs -xv svc:/system/intrd:default' to determine the
generic reason why the service failed, the location of any logfiles,
and a list of other services impacted.
root@openindiana:~# mv /usr/lib/intrd.disabled /usr/lib/intrd
root@openindiana:~# svcadm clear intrd
root@openindiana:~# fmadm faulty
root@openindiana:~#
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment