An inline input parser for timestamped and filtered output

filterstamp

#!/usr/bin/env bash

set -eu


########################################
## Introduction
########################################
HELP_INFO=$(cat <<'CONTENTS_HEREDOC'
input_filter v0.1
Copyright (c) 2019, Matt Johnson (matt.johnson@classyllama.com). All 
rights reserved.

An inline input parser for timestamped and filtered output
https://gist.github.com/mttjohnson/417b8c4f1b0beff8767f2df0f478e3b5

This script expects input to be piped into it, will filter the input and 
inclue a timestamp as a prefix to each line of the output. If not filters
are provided as options, the output is just timestamped.

Usage: filterstamp [options]
  -t=text, --filter-text=text     Text value of line to filter.
  -f=file, --filter-file=file     Path to file containing lines to filter.

Examples:
  # Just timestamp the output
  echo "some content" | filterstamp
  
  # Supply filter for a single line value to be excluded from output
  echo "some content" | filterstamp -t="some content"
  
  # Supply multiple lines that should be removed from the output
  echo "some content" | filterstamp -f=filter_list.txt
  
  # Filter out multiple lines and any empty lines from the output
  echo "some content" | filterstamp -e -f=filter_list.txt
  
  # Filter multiple line from output and send to log file in addition
  # to stdout. Useful for crontab entries to be filtered and logged.
  echo "some content" | filterstamp -f=filter_list.txt \
    | tee -a content_filtered.log

CONTENTS_HEREDOC
)

# Check to see if a pipe exists on stdin.
if ! [ -p /dev/stdin ]; then
  # if /dev/stdin is not a pipe
  echo "${HELP_INFO}"
  exit;
fi


########################################
## Command Line Options
########################################
declare FILTER_TEXT=""
declare FLAG_FILTER_TEXT=false
declare FILTER_FILE=""
declare FLAG_FILTER_FILE=false
declare FLAG_FILTER_EMPTY_LINES=false
for switch in $@; do
    case $switch in
        -t=*|--filter-text=*)
            FILTER_TEXT="${switch#*=}"
            if [[ ! "${FILTER_TEXT}" =~ ^.+$ ]]; then
                >&2 echo "Error: Invalid value given -t|--filter-text=${FILTER_TEXT}"
                exit -1
            fi
            FLAG_FILTER_TEXT=true
            ;;
        -f=*|--filter-file=*)
            FILTER_FILE="${switch#*=}"
            if [[ ! "${FILTER_FILE}" =~ ^.+$ ]]; then
                >&2 echo "Error: Invalid value given -f|--filter-file=${FILTER_FILE}"
                exit -1
            fi
            if ! [ -f "${FILTER_FILE}" ]; then
                >&2 echo "Error: No file found at location specified"
                exit -1
            fi
            FLAG_FILTER_FILE=true
            ;;
        -e|--filter-empty-lines)
            FLAG_FILTER_EMPTY_LINES=true
            ;;
        *)
            echo "${HELP_INFO}"
            exit;
            ;;
    esac
done


########################################
## Build list of lines to filter
########################################
declare -a LINES_TO_FILTER_OUT=()

if [[ ${FLAG_FILTER_TEXT} == true ]]; then
    LINES_TO_FILTER_OUT+=("${FILTER_TEXT}")
fi

if [[ ${FLAG_FILTER_FILE} == true ]]; then
  # Read each line from file and add to filter array
  while IFS='' read -r LINE_FROM_FILE || [[ -n "${LINE_FROM_FILE}" ]]; do
    LINES_TO_FILTER_OUT+=("${LINE_FROM_FILE}")
  done < "${FILTER_FILE}"
fi


########################################
## Define Line Prefix
########################################
TIMESTAMP=$(date "+%Y-%m-%d %H:%M:%S %Z")
LINE_PREFIX="[${TIMESTAMP}]: "


########################################
## Filter piped in contents
########################################
while IFS= read line; do
  MATCH_COUNT=0
  
  # Loop through the array of filter lines
  if [ "${#LINES_TO_FILTER_OUT[@]}" -gt "0" ]; then
    for EACH_FILTER in "${LINES_TO_FILTER_OUT[@]}"
    do
      # When the input line matches a filter line, incremet the MATCH_COUNT variable
      if [ "${line}" == "${EACH_FILTER}" ]; then MATCH_COUNT=$((MATCH_COUNT+1)); fi
    done
  fi
  
  # Check for empty lines
  if [[ ${FLAG_FILTER_EMPTY_LINES} == true ]]; then
      if [ "${line}" == "" ]; then MATCH_COUNT=$((MATCH_COUNT+1)); fi
  fi
  
  # Only output line that did not match any filters
  if [ "${MATCH_COUNT}" -eq "0" ]; then
    # Include prefix in addition to the input line
    echo "${LINE_PREFIX}${line}"
  fi
  
done

I used this as means of wrapping around some program that I couldn't change so that any output included timestamp information, and allowed me to filter out certain output from the program that I didn't want to log.

It can be frustrating looking through a log file to troubleshoot some behavior and not have any reference to when the output in the log file was produced.

Using this in a crontab allowed me to filter out output from the program that I didn't want to receive an email notification about since any command on a crontab that produces output can be emailed.

  # Filter multiple line from output and send to log file in addition
  # to stdout. Useful for crontab entries to be filtered and logged.
  echo "some content" | filterstamp -f=filter_list.txt \
    | tee -a content_filtered.log