Created on September 24, 2018
- 1. Setup
- 2. Configure Sensu
- 3. Create a pipeline to send alerts to Slack
- 4. Create a check to monitor Nginx
- 5. Create a check to monitor CPU usage
Download the Sensu Skunkworks repository.
# Download the virtual machine
cd Documents && git clone git@github.com:sensu/skunkworks.git
Install the Vagrant virtual machine running the CentOS operating system and Sensu Core Edition:
# Install the virtual machine
cd skunkworks/sensu-ce-vagrant-centos && vagrant up
To see what it's doing, check out https://github.com/sensu/skunkworks/blob/master/sensu-ce-vagrant-centos/sensu2-centos.sh.
Note: You can use vagrant destroy then vagrant up to erase and restart the virtual machine.
If you're already an admin of a Slack, visit https://YOUR WORKSPACE NAME HERE.slack.com/services/new/incoming-webhook and follow the steps to add the Incoming WebHooks integration, choose a channel, and save the settings.
(If you're not yet a Slack admin, start here to create a new workspace.)
After saving, you'll see your webhook URL under Integration Settings.
Once the install process has completed, we can access the virtual machine.
# Access the virtual machine
vagrant ssh
You should see your command prompt change to indicate that you're accessing the virtual machine.
Note: To exit out of the virtual machine, use CTRL+D.
You should see a line in your terminal that starts with Access the dashboard at.
Copy and paste that URL into your browser and sign in as the default admin user
(username:admin, password:P@ssw0rd!).
Use Nano to open the Sensu agent configuration file and set the backend URL and add a subscription.
# Open the agent configuration file
sudo nano /etc/sensu/agent.yml
Look for these lines:
#subscriptions: ""
#backend-url:
# - "ws://127.0.0.1:8081"
Remove the # and add a subscription, so it looks like:
subscriptions: "testing"
backend-url:
- "ws://127.0.0.1:8081"
To save and exit nano, CTRL+X then Y then ENTER.
Now we'll start the Sensu agent:
# Start the Sensu agent
sudo systemctl start sensu-agent
sensuctl configure
Enter the following information when prompted:
? Sensu Backend URL: http://127.0.0.1:8080
? Username: admin
? Password: P@ssw0rd!
? Organization: default
? Environment: default
? Preferred output format:
❯ none
json
wrapped-json
To get started, let's create a pipeline to send alerts to Slack. We've already installed this Sensu Slack handler during setup. We can test it using:
# Access help view for Sensu Slack handler
slack-handler -h
Let's create a Slack handler. Copy the following into a text editor and swap in your webhook URL.
sensuctl handler create slack \
--type pipe \
--command 'slack-handler \
--webhook-url https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX \
--channel monitoring' \
--filters is_incident,not_silenced
Verify it using:
# Do I have any pipelines set up?
sensuctl handler list
Nginx is a webserver that we installed and started during setup. You can make sure it's running using:
curl -I http://localhost:80
First off, we'll create a check to monitor Nginx.
sensuctl check create check-nginx \
--command 'http_check.sh http://localhost:80' \
--interval 20 \
--subscriptions testing \
--handlers slack
See the events created by this check:
# What events is Sensu receiving?
sensuctl event list
Now we'll stop Nginx so that we receive an alert from Sensu.
sudo systemctl stop nginx
You should see an alert in Slack. Start Nginx to resolve the alert.
sudo systemctl start nginx
Note: Due to a bug in the way Sensu works right now, you won't see a resolution notification in Slack.
As part of setup, we installed a check to monitor CPU usage. You can test it using:
# Check CPU usage
check-cpu.sh
To create a check that monitors CPU usage:
sensuctl check create check-cpu \
--command 'check-cpu.sh -w 75 -c 90' \
--interval 20 \
--subscriptions testing \
--handlers slack
And verify with:
# What events is Sensu receiving?
sensuctl event list
To test this check, we'll use a program called Stress that we installed during setup.
# Stress my CPU
stress -c 1
We should get an alert in Slack while Stress is running.
Quit stress with CTRL+C to resolve the alert.
Note: Due to a bug in the way Sensu works right now, you won't see a resolution notification in Slack.