-
-
Save seth-paxton/d19571620e8986c2ef1a to your computer and use it in GitHub Desktop.
| 1. Download the plugin from: https://github.com/sensu/sensu-community-plugins/blob/master/handlers/remediation/sensu.rb | |
| 2. Create the handler definition: | |
| { | |
| "handlers": { | |
| "remediator": { | |
| "type": "pipe", | |
| "command": "/etc/sensu/handlers/remediator.rb" | |
| } | |
| } | |
| } | |
| 3. Create check definition: | |
| { | |
| "checks": { | |
| "seyren_check": { | |
| "command": "/etc/sensu/plugins/check-procs.rb -p PROC", | |
| "interval": 10, | |
| "subscribers": ["SUB"], | |
| "handlers": ["remediator","pagerduty"], | |
| "occurrences": 1, | |
| "refresh": 10, | |
| "remediation": { | |
| "light_remediation": { | |
| "occurrences": [1, 2], | |
| "severities": [1] | |
| }, | |
| "medium_remediation": { | |
| "occurrences": ["3-10"], | |
| "severities": [1] | |
| }, | |
| "heavy_remediation": { | |
| "occurrences": ["1+"], | |
| "severities": [2] | |
| } | |
| } | |
| }, | |
| "light_remediation": { | |
| "command": "touch /tmp/test", | |
| "subscribers": [], | |
| "handlers": ["pagerduty"], | |
| "pager_team": "testing", | |
| "interval": 10, | |
| "publish": false | |
| }, | |
| "medium_remediation": { | |
| "command": "touch /tmp/test", | |
| "subscribers": [], | |
| "handlers": ["pagerduty"], | |
| "pager_team": "testing", | |
| "interval": 10, | |
| "publish": false | |
| }, | |
| "heavy_remediation": { | |
| "command": "touch /tmp/test", | |
| "subscribers": [], | |
| "handlers": ["pagerduty"], | |
| "pager_team": "testing", | |
| "interval": 10, | |
| "publish": false | |
| } | |
| } | |
| } | |
| 4. I noticed in the Sensu-API log that remediator was POSTing a check to the API with the subscribers being the hostname of the affected server. Sample log message: | |
| {"timestamp":"2014-10-06T18:46:16.656712+0000","level":"info","message":"POST /request","remote_address":"127.0.0.1","user_agent":"Ruby","request_method":"POS | |
| T","request_uri":"/request","request_body":"{\"check\":\"heavy_remediation\",\"subscribers\":[\"hostname.example.com\"]}"} | |
| 5. I changed my test client to subscribe to that queue and success. Everything works now: | |
| { | |
| "client":{ | |
| "subscriptions":[ | |
| "test", | |
| "hostname.example.com" | |
| ], | |
| "name":"hostname.example.com", | |
| "address":"555.555.555.555" | |
| } | |
| } |
Agreed with above comment! this shall create unnecessary 'subscription' which occupies Redis backend and also clutters the subscriptions, so also additional update to the client.json ifile !
What could be other possible way out! I wish Sensu API should have a way to push the checks for only failing nodes...
While not ideal, I believe that by subscribing to the hostname, this is the only way remediation will actually work.
+1 anything new on this?
To avoid cluttering your subscriptions with hostnames, you could put the following line in your seyren_check definition, right after the remediation value:
"trigger_on": ["SUB"]
This will trigger the remediation commands on all clients that subscribe to SUB
Obviously, this solution will only work properly if one of the following conditions is met:
- your remediation command is non-destructive
- your remediation command has some error-checking built in and only runs if the error condition is met
- you only have one server subscribed
Otherwise your remediation command may fail or interrupt services or do other bad things.
hi, adding the hostname to the subscription will clutter the dashboard. Every client will be a subscription in the frontend. Is there a better approach?