Kill idle RabbitMQ connections

kill-idle-rabbitmq-connections.sh

#!/usr/bin/env bash

check_connection() {
  PID=$(echo "$1" | awk '{print $1}')
  SOURCE=$(echo "$1" | awk '{print $2}')

  SOURCE_STATUS=$(curl --silent --fail "$SOURCE" --max-time 1 > /dev/null)

  # Status code returned when a timeout occurs
  if [[ $? -eq 28 ]]; then
    echo "connection seems down, closing $PID"
    rabbitmqctl close_connection --quiet --silent "$PID" "close"
  fi
}

export -f check_connection

echo "Checking connections..."

while true; do

  CONNECTIONS=$(rabbitmqctl list_connections --no-table-headers --quiet pid name)
  echo "$(echo "$CONNECTIONS" | wc -l) connections were found."
  echo "$CONNECTIONS" | xargs -P 20 -I {} /bin/bash -c 'check_connection "{}"'

  echo "Done checking connections."

  sleep 600
done

This script is used in the RabbitMQ Helm release to find and close idle connections due to Celery not properly opening connections with the given heartbeat (see: celery/celery#7250).

If the ip:port returns a "connection refused", we can assume that at least, the container/pod is still running and the worker is OK. If it time outs, it means the IP is not even used anymore by Kubernetes for a pod/worker therefore the RMQ connection is useless, we can close it.

It is needed as the growing of RMQ connections makes the RAM explode.