"Sqlite3: Sleeping for 200ms to retry busy DB" - Plex BusyDB Check, Restart and Notify Script with Apprise Notifications (predefined Telegram) and statistics (currently testing)

plex_busydb_checkv3_restart_and_ notify.sh

#!/bin/bash

# Files and paths
timestamp_file="/tmp/plex_busy_db_timestamp"
log_file="/tmp/plex_events.log"
lock_file="/tmp/plex_script.lock"
restart_grace_file="/tmp/plex_restart_grace"

# Apprise URL for Telegram notifications
APPRIZE_URL="tgram://bot_token/A,B/"

# Grace period in seconds (3 minutes)
grace_period=180

# Locking mechanism
if [ -e "$lock_file" ]; then
    echo "Another instance of the script is running."
    exit 1
else
    touch "$lock_file"
fi

# Cleanup lock on exit
trap 'rm -f "$lock_file"' EXIT

# Check for grace period after a restart
if [ -e "$restart_grace_file" ]; then
    last_restart=$(cat "$restart_grace_file")
    current_time=$(date +%s)
    time_since_restart=$((current_time - last_restart))

    if [ "$time_since_restart" -lt "$grace_period" ]; then
        echo "In grace period after a restart. Skipping checks."
        exit 0
    else
        rm "$restart_grace_file"
    fi
fi

# Function to send a notification using Apprise
send_notification() {
    local title="$1"
    local body="$2"
    apprise -t "$title" -b "$body" $APPRIZE_URL
    if [ $? -ne 0 ]; then
        echo "Error: Failed to send notification via Apprise."
        exit 1
    fi
}

# Function to log events
log_event() {
    local event_type="$1"
    echo "$(date '+%Y-%m-%d %H:%M:%S') $event_type" >> "$log_file"
    if [ $? -ne 0 ]; then
        send_notification "Script Error" "Failed to write to log file."
        exit 1
    fi
}

# Function to cleanup old log entries (older than 7 days)
cleanup_logs() {
    if [ ! -f "$log_file" ]; then
        return 0
    fi
    
    local seven_days_ago=$(date -d "7 days ago" '+%Y-%m-%d')
    awk -v d="$seven_days_ago" '{split($1, date, "-"); if (date[1] "-" date[2] "-" date[3] >= d) print $0}' "$log_file" > "${log_file}.tmp"
    if [ $? -ne 0 ]; then
        send_notification "Script Error" "Failed during log cleanup."
        exit 1
    fi
    mv "${log_file}.tmp" "$log_file"
}

# Function to calculate and send a 12-hour statistics summary
send_12h_statistics() {
    local now=$(date +%s)
    local twelve_hours_ago=$((now - 12*3600))
    local summary=$(awk -v d="$twelve_hours_ago" 'BEGIN {count=0} {split($1 " " $2, datetime, "-| |:"); event_time=mktime(datetime[1] " " datetime[2] " " datetime[3] " " datetime[4] " " datetime[5] " " datetime[6]); if (event_time > d) print $0}' "$log_file")
    send_notification "Plex 12-hour Statistics" "$summary"
}

# Function to check the health of the plex container
check_plex_health() {
    local plex_logs=$(docker logs --tail 10 plex)
    if [ $? -ne 0 ]; then
        send_notification "Script Error" "Failed to retrieve Plex container logs."
        exit 1
    fi

    local count=$(grep -c "Sqlite3: Sleeping for 200ms to retry busy DB" <<< "$plex_logs")
    if [ $? -ne 0 ]; then
        send_notification "Script Error" "Failed during log processing."
        exit 1
    fi

    if [ "$count" -gt 4 ]; then
        log_event "busy_db"
        docker restart plex
        if [ $? -ne 0 ]; then
            send_notification "Script Error" "Failed to restart Plex container."
            exit 1
        fi
        log_event "restart"
        send_notification "Plex Container Restarted" "Container was restarted due to database issues."
        echo $(date +%s) > "$restart_grace_file"
    fi

    local last_line=$(docker logs --tail 1 plex)
    if [ $? -ne 0 ]; then
        send_notification "Script Error" "Failed to retrieve last line of Plex container logs."
        exit 1
    fi

    if grep -q "Sqlite3: Sleeping for 200ms to retry busy DB" <<< "$last_line"; then
        if [ -f "$timestamp_file" ]; then
            local current_time=$(date +%s)
            local last_time=$(cat "$timestamp_file")
            local time_diff=$((current_time - last_time))
            if [ "$time_diff" -gt 90 ]; then
                log_event "busy_db"
                docker restart plex
                if [ $? -ne 0 ]; then
                    send_notification "Script Error" "Failed to restart Plex container due to busy DB event."
                    exit 1
                fi
                log_event "restart"
                send_notification "Plex Container Restarted" "Container was restarted due to last line being busy DB for more than 90 seconds."
                rm "$timestamp_file"
            fi
        else
            date +%s > "$timestamp_file"
        fi
    else
        [ -f "$timestamp_file" ] && rm "$timestamp_file"
    fi
}

# Run the health check function
check_plex_health

# Call the send_12h_statistics function at 10am and 10pm
current_hour=$(date +'%H')
if [ "$current_hour" -eq "10" ] || [ "$current_hour" -eq "22" ]; then
    send_12h_statistics
fi

# Cleanup logs older than 7 days
cleanup_logs

Plex BusyDB Check, Restart and Notify Script Documentation

Overview:

This script checks the logs of a Docker container running Plex to detect database "busy" errors. If the script detects multiple "busy" messages in a short span or if the last message in the logs indicates a "busy" database for more than 90 seconds, it will restart the Plex container. This is currently the only "fix" known to me to unfreeze Plex, besides mitigating the problem through high-end hardware (waiting for it to resolve itself just doesn't happen). The script also provides a daily statistics summary of events and ensures that the script operations are atomic to prevent overlaps when run frequently.

Intended Purpose:

• Monitor Plex logs for "busy" database messages.
• Restart the Plex container under certain conditions to resolve potential "busy" database issues.
• Send notifications when the container is restarted.
• Provide a daily statistics summary of events.

Prerequisites:

1. Docker with Plex running as a container.
2. Apprise installed for sending notifications.
3. The script requires execution permissions (chmod +x plex_busydb_check.sh).

Setup:

1. Clone/download the script to your server or machine running the Plex Docker container.
2. Modify the APPRIZE_URL placeholder in the script with your actual Apprise URL. This is used to send notifications. For instance, for Telegram, it would look like: tgram://bot_token/CHAT_ID/.
3. Make sure the script has execute permissions:

chmod +x plex_busydb_check.sh

4. Run the script manually or set it up in a cron job to run at your desired frequency (recommended every minute).

Usage:

Run the script:

./plex_busydb_check.sh

To set it up as a cron job to run every minute:

* * * * * /path/to/plex_busydb_check.sh

Notifications:

The script sends notifications in the following scenarios:

1. When the Plex container is restarted due to database issues.
2. If there's an error while the script is executing.
3. A daily summary of the events.

Notes:

• The script maintains logs in the /tmp directory by default. Ensure this location is writable by the user executing the script or modify the script to change the directory.
• Monitor the initial runs of the script to ensure it's working as expected in your environment.