Cleans up duplicate go2rtc ffmpeg ingestion processes

README.md

ffmpeg Duplicate Process Terminator

This script checks for duplicate ffmpeg processes running on the system and sends a SIGHUP signal to the unwanted ones to terminate them. If a process does not terminate after a certain timeout, the script sends a SIGKILL signal to force the termination.

The script lists out the processes using ps -aux command and identifies the ffmpeg processes. The script then groups the processes by their stream id (the streams are initiated to create/encode video feeds) and send termination signals to any duplicate processes.

Usage

The script can be used with optional command-line argument -t or --timeout which specifies the timeout interval in seconds after which a SIGKILL signal is sent if the process is still running after receiving a SIGHUP signal. If no timeout is specified, it defaults to 15 seconds or uses the environment variable FFMPEG_MONITOR_HUP_TIMEOUT if set.

./ffmpeg_duplicate_process_terminator.py [-t TIMEOUT]

If the -t option is used with an environment variable, the command line argument is favored.

Dependencies

This script is written in Python3 and uses the following modules:

• os
• subprocess
• time
• argparse
• signal
• re
• errno

Key Functions

• parse_args(): Parses command-line arguments.
• get_timeout(args): Retrieves timeout setting from the command-line arguments or environment variable.
• ps_ef(): Executes ps -aux command to get the detailed snapshot of the current running processes.
• get_ffmpeg_processes(ps_ef_output): Gets ffmpeg processes with stream_id from ps -aux output.
• group_processes_by_stream_id(processes): Groups process ids by stream id.
• is_process_running(pid): Checks if a process is still running.
• kill_duplicate_processes(streams, timeout): Kills duplicate ffmpeg processes.
• main(): Acts as the entry point when the script is run.

Exit Status

If no ffmpeg processes are running, the script raises a warning.

Warning

The script should be run by a user with privileges to kill processes.

Note: The processes are killed based on the assumption that the only duplicate ffmpeg processes running would be unwanted. Please only execute the script if you're sure there are no other necessary ffmpeg processes running on your system.

monitor-go2rtc.sh

#!/usr/bin/env python3

import os
import subprocess
import time
import argparse
import signal
import re
import errno

def parse_args():
    """
    Parses command-line arguments
    :return: Namespace with command-line argument values as its attributes.
    """
    parser = argparse.ArgumentParser(
        description='This script checks for duplicate ffmpeg processes and sends \
        a SIGHUP signal to the unwanted ones. If a process does not terminate \
        after a certain timeout, the script sends a SIGKILL signal to force termination.') 
    parser.add_argument('-t', '--timeout', type=int, 
                        help='Timeout interval in seconds after which a SIGKILL signal is sent if the process \
                        is still running after receiving a SIGHUP signal.')
    return parser.parse_args()

def get_timeout(args):
    """
    Retrieves timeout setting from the command-line arguments or environment variable.
    :param args: Namespace with the command-line argument values as attributes.
    :return: Timeout interval in seconds.
    """
    if args.timeout:
        timeout = args.timeout
        if 'FFMPEG_MONITOR_HUP_TIMEOUT' in os.environ:
            print('Warning: both command-line argument and environment variable \
                  FFMPEG_MONITOR_HUP_TIMEOUT are set. Value from the command-line argument will be used.')
    elif 'FFMPEG_MONITOR_HUP_TIMEOUT' in os.environ:
        timeout = int(os.getenv('FFMPEG_MONITOR_HUP_TIMEOUT'))
    else:
        timeout = 15

    return timeout 

def ps_ef():
    """
    Executes `ps -aux` command to get the detailed snapshot of the current running processes.
    :return: Output string from `ps -aux` command.
    """
    return subprocess.check_output(['ps', 'aux']).decode('utf-8')

def get_ffmpeg_processes(ps_ef_output):
    """
    Gets ffmpeg processes with stream_id from ps -ef output.
    :param ps_ef_output: string output from ps -ef command.
    :return: list of tuples where the first element is the process id and the second element is the stream id.
    """
    pattern = re.compile(r' (\d+) .*ffmpeg.*-f rtsp rtsp://127.0.0.1:8554/([a-fA-F0-9]{32})\s*$', re.MULTILINE)
    return pattern.findall(ps_ef_output)

def group_processes_by_stream_id(processes):
    """
    Groups process ids by stream id.
    :param processes: List of tuples (process id, stream id)
    :return: Dictionary where keys are stream ids and values are lists with process ids.
    """
    streams = {}
    for pid, stream_id in processes:
        pid = int(pid)
        if stream_id not in streams:
            streams[stream_id] = []
        streams[stream_id].append(pid)
    return streams

def is_process_running(pid):
    """
    Checks if a process is running.
    :param pid: Integer process id.
    :return: Boolean value stating if process is running or not.
    """
    try:
        os.kill(pid, 0)
    except OSError as ex:
        return ex.errno != errno.ESRCH

def kill_duplicate_processes(streams, timeout):
    """
    Kills duplicate ffmpeg processes
    :param streams: Dictionary where keys are stream ids, and values are list of process ids.
    :param timeout: Interval in seconds the script waits after sending a SIGHUP before it sends a SIGKILL signal.
    """
    for stream_id, pids in streams.items():
        if len(pids) > 1:
            for pid in sorted(pids)[:-1]:
                # Try to send SIGHUP signal first
                try:
                    os.kill(pid, signal.SIGHUP)
                    print(f'Sent SIGHUP to process {pid}(stream {stream_id})')
                except OSError:
                    print(f'Cannot send SIGHUP signal to process {pid}({stream_id}), skipping')
                
                # Then wait for the process to finish
                time.sleep(timeout)
                if is_process_running(pid):
                    # If not finished, send SIGKILL signal
                    try:
                        os.kill(pid, signal.SIGKILL)
                        print(f'Sent SIGKILL to {pid}(stream {stream_id}) as it did not terminate after {timeout} seconds')
                    except OSError:
                        print(f'Cannot send SIGKILL signal to process {pid}(stream {stream_id}), skipping')

def main():
    args = parse_args()
    timeout = get_timeout(args)
    ps_ef_output = ps_ef()
    
    ffmpeg_processes = get_ffmpeg_processes(ps_ef_output)

    if not ffmpeg_processes:
        print('Warning: there are no ffpmeg processes running.')
        return

    streams = group_processes_by_stream_id(ffmpeg_processes)
    kill_duplicate_processes(streams, timeout)

if __name__ == "__main__":
    main()