Skip to content

Instantly share code, notes, and snippets.

@gilangvperdana

gilangvperdana/README.md

Last active May 19, 2022 09:54
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Embed
What would you like to do?
Learn more about clone URLs
Download ZIP
Systemd Alert to Telegram
Raw
README.md

Systemd Alert to Telegram

The purposes is to auto alerting while systemd goes to failed/activate/stopped state to telegram with efficient scripting & services.

Warn

your_service is your systemd service who you want to monitor.

Configure

  • Set Telegram Key
mkdir /etc/telegram
nano /etc/telegram/key.sh
  • TELEGRAM_BOT_KEY you can get from BOT_FATHER on telegram
  • CHAT_ID you can get from https://api.telegram.org/botYOUR_TELEGRAM_BOT_KEY/getUpdates
export TELEGRAM_KEY="TELEGRAM_BOT_KEY"
export CHAT_ID="-1001795547276" 

chmod +x /etc/telegram/key.sh
  • Create Telegram Service
nano /usr/bin/telegram

#!/bin/bash
source /etc/telegram/key.sh

URL="https://api.telegram.org/bot5136513143:AAGscRDJ_t5W3Al54vlhbLxnXBl52WD85lQ/sendMessage"
curl -s -d "chat_id=$CHAT_ID&disable_web_page_preview=1&text=$1" $URL > /dev/null

chmod +x /usr/bin/telegram
  • Create Script for All State of Services, we can make Activating|Failed|Stopped State in one Script
nano /usr/bin/your_service-status-allinone-telegram

#!/bin/bash

shopt -s nocasematch

UNIT=your_service_you_want_to_monitor
status=$(systemctl is-active $UNIT)
HOSTNAME=$(hostname)
IPPRIV=$(ifconfig ens3 | grep "inet " | awk '{print $2}')
UNITSTATUS=$(systemctl status $UNIT | grep 'Active')
ALERT=$(echo -e "\u26A0")
DATE=$(date)

case $status in

    active)
    telegram "$ALERT Alert Report $UNIT Active $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV

    Message:
    $UNITSTATUS

    $UNIT currently in active state"
    ;;

    activating)
    telegram "$ALERT Alert Report Elasticsearch Activating $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV

    Message:
    $UNITSTATUS

    $UNIT currently in activating state"
    ;;

    loaded)
    telegram "$ALERT Alert Report Elasticsearch Loaded $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV

    Message:
    $UNITSTATUS

    $UNIT currently in loaded state"
    ;;

    failed)
    telegram "$ALERT Alert Report Elasticsearch Failed $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV

    Message:
    $UNITSTATUS

    Ops, $UNIT is on failed state. Please check cluster now!"
    ;;

    deactivating)
    telegram "$ALERT Alert Report Elasticsearch Deactivating $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV

    Message:
    $UNITSTATUS

    Ops, $UNIT is on deactivating state. Please check cluster now!"
    ;;

    inactive)
    telegram "$ALERT Alert Report Elasticsearch Inactive $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV

    Message:
    $UNITSTATUS

    Ops, $UNIT is on Inactive state. Please check cluster now!"
    ;;

esac

chmod +x /usr/bin/your_service-status-allinone-telegram
  • Make Script can Execute from Systemd-Journald
nano /etc/systemd/system/your_service-status-failed-telegram@.service

[Unit]
Description=Unit Status Telegram Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/your_service-status-allinone-telegram %I
  • Declare to Systemd Service, you can check the systemd directory from systemctl status your_service
[Unit]
OnFailure=your_service-status-allinone-telegram@.service

[Service]
ExecStopPost=+/usr/bin/your_service-status-allinone-telegram 
ExecStartPost=+/usr/bin/your_service-status-allinone-telegram

chmod +x /etc/systemd/system/your_service-status-failed-telegram@.service
  • Daemon Reload
systemctl daemon-reload
  • Test
systemctl restart your_service
systemctl stop your_service
systemctl start your_service

[UPDATE] Efficiency Systemd Alert to Telegram

This version just use one systemd service and script for all service who want to monitor. The difference is only in the addition of parameters after command execution, eg ./systemd-alert your_service

Warn

your_service is your systemd service who you want to monitor.

Configure

  • Set Telegram Key
mkdir /etc/telegram
nano /etc/telegram/key.sh
  • TELEGRAM_BOT_KEY you can get from BOT_FATHER on telegram
  • CHAT_ID you can get from https://api.telegram.org/botYOUR_TELEGRAM_BOT_KEY/getUpdates
export TELEGRAM_KEY="TELEGRAM_BOT_KEY"
export CHAT_ID="-1001795547276" 

chmod +x /etc/telegram/key.sh
  • Create Telegram Service
nano /usr/bin/telegram

#!/bin/bash
source /etc/telegram/key.sh

URL="https://api.telegram.org/bot5136513143:AAGscRDJ_t5W3Al54vlhbLxnXBl52WD85lQ/sendMessage"
curl -s -d "chat_id=$CHAT_ID&disable_web_page_preview=1&text=$1" $URL > /dev/null

chmod +x /usr/bin/telegram
  • Create Script for All State of Services, we can make Activating|Failed|Stopped State in one Script
nano /usr/bin/alerting-systemd

#!/bin/bash

shopt -s nocasematch

UNIT=$1
status=$(systemctl is-active $UNIT)
HOSTNAME=$(hostname)
IPPRIV=$(ifconfig ens3 | grep "inet " | awk '{print $2}')
IPPUB=$(curl ifconfig.me)
UNITSTATUS=$(systemctl status $UNIT | grep 'Active')
ALERT=$(echo -e "\u26A0")
DATE=$(date)

case $status in

    active)
    telegram "[Active] $UNIT on Active State $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV
    Public IP: $IPPUB

    Message:
    $UNITSTATUS

    $UNIT currently in active state"
    ;;

    activating)
    telegram "[ACTIVATING] $UNIT on Activating State $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV
    Public IP: $IPPUB

    Message:
    $UNITSTATUS

    $UNIT currently in activating state"
    ;;

    loaded)
    telegram "[LOADED] $UNIT on Loaded State $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV
    Public IP: $IPPUB

    Message:
    $UNITSTATUS

    $UNIT currently in loaded state"
    ;;

    failed)
    telegram "[FAILED] $UNIT on Failed State $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV
    Public IP: $IPPUB

    Message:
    $UNITSTATUS

    Ops, $UNIT is on failed state. Please check cluster now!"
    ;;

    deactivating)
    telegram "[DEACTIVATING] $UNIT on Deactivatiing State $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV
    Public IP: $IPPUB

    Message:
    $UNITSTATUS

    Ops, $UNIT is on deactivating state. Please check cluster now!"
    ;;

    inactive)
    telegram "[INACTIVE] $UNIT on Inactive State $ALERT

    Date: $DATE
    Host: $HOSTNAME
    IP: $IPPRIV
    Public IP: $IPPUB

    Message:
    $UNITSTATUS

    Ops, $UNIT is on Inactive state. Please check cluster now!"
    ;;

esac

chmod +x /usr/bin/alerting-systemd
  • Make Script can Execute from Systemd-Journald
nano /etc/systemd/system/alerting-systemd@.service

[Unit]
Description=Unit Status Telegram Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/alerting-systemd $1 %I

chmod +x /etc/systemd/system/alerting-systemd@.service
  • Declare to Systemd Service, you can check the systemd directory from systemctl status your_service
[Unit]
OnFailure=alerting-systemd@.service `your_service`

[Service]
ExecStopPost=+/usr/bin/alerting-systemd `your_service`
ExecStartPost=+/usr/bin/alerting-systemd `your_service`
  • Daemon Reload
systemctl daemon-reload
  • Test
systemctl restart your_service
systemctl stop your_service
systemctl start your_service
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment