Skip to content

Instantly share code, notes, and snippets.

@mwvent
Last active September 23, 2021 15:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save mwvent/f4d305996756336ff85d6f5ea1fb4c76 to your computer and use it in GitHub Desktop.
Save mwvent/f4d305996756336ff85d6f5ea1fb4c76 to your computer and use it in GitHub Desktop.
Zoneminder Record High-res H264 streams 24/7 with low CPU Load
>>>>>>>>>>>>>>>>>>>>> BIG UPDATE <<<<<<<<<<<<<<<<<<<<<<<<<<<<
On zoneminder version 1.35.16 and later there is a DecodingEnabled setting for each monitor - this script is therefore of little use anymore
>>>>>>>>>>>>>>>>>>>>> ORIGINAL README <<<<<<<<<<<<<<<<<<<<<<<<<<<<
The original Zoneminder Forum thread about all this is here:
https://forums.zoneminder.com/viewtopic.php?f=9&t=27537&p=107235#p107235
Many thanks to Matt Watts for his improvements.
Notes.
You will need to install mediainfo (apt-get install mediainfo).
By default, zmrecord is set to save the continuous recordings in 600
second (10 minute) chunks. Change the value of RECORD_TIME at the top
of zmrecord.sh to alter this.
Test
sudo zmrecord.sh <Monitor_Id>
and you should see recording files appearing in <Monitor Storage Path>/<Monitor-ID>
eg if Monitor 7's Storage path is /mnt/bigraid/zoneminder/recordings the
mp4 files should appear in:
/mnt/bigraid/zoneminder/recordings/events/7/<event-id>/...
Installation
sudo cp zmrecord.sh /opt/zmrecord.sh
sudo chmod 0554 /opt/zmrecord.sh
sudo cp zmrecord@.service /lib/systemd/system
Then for the monitors you wish to record do:
systemctl enable zmrecord@<MonitorID..>.service
systemctl start zmrecord@<MonitorID..>.service
eg for Monitors 2,3 and 6
systemctl enable zmrecord@2.service zmrecord@3.service zmrecord@6.service
systemctl start zmrecord@2.service zmrecord@3.service zmrecord@6.service
#!/bin/bash
# Original script posted here https://forums.zoneminder.com/viewtopic.php?f=9&t=27537 by russell_i_brown
# Install to /opt/zmrecord.sh
# Grab a video stream from an IP camera into a temp file.
# Inject an 'Events' record into the ZM Database.
# Create the directory in the zm storage area
# Move the mp4 to the right place
#
# Call with the Camera Id number as arg1:
#
# zmrecord.sh 12
#
#
# You might have to install mediainfo which I use to grab
# the frame count.
#
# Version 0.1 - russell_i_brown 18/ 9/18 First hack
# Version 0.2 - russell_i_brown 19/ 9/18 Read monitor guff from DB
# Version 0.3 - russell_i_brown 19/ 9/18 Set StorageId in DB
# Version 0.4 - matt_watts 10/12/19 Use FFMPEG segment output to ensure frames are not missed between segments
# Version 0.5 - matt_watts 12/12/19 Move video segments immediatley into database / storage tree while live and keep db stats updated
# Version 0.6 - matt_watts 16/12/19
# - Replace FFMPEG list output with simple folder polling allowing some crash/script kill recovery
# - Use FFMPEG empty_moov flag to allow playing in-recording segment in the web ui and preventing
# corrupt video on crash/script kill
# - Continuous event database updates as segment is being wrote
# - Log some errors and events direct to Zoneminders DB Log
# Version 0.7 - russell_i_brown 31/01/20 Use Event Prefix as requested by tuxmos
# Handle Monitor with a StorageId of 0 by defaulting to 1 if blank (ZM 1.34 issue?)
# Calculate the Bandwidth so the ZM console looks sane.
# Version 0.8 - ZM Update has changes Events.StartTime to Events.StartDateTime and Events.EndTime to Events.EndDate time
# otherwise same as 0.7
#########################################
# Recording chunk time in seconds
# This script will record for this number of seconds at a time
# 600 (10 minutes) is what I use.
RECORD_TIME=600
#
#########################################
trap "kill 0" SIGINT
# ZM Log
function zm_log() {
CODE="$1"
CODEVAL="$2"
MESSAGE="`echo $3|tr -d '"'`"
TIMESTAMP=`date +%s`
SQL="INSERT INTO Logs ( Component,File,ServerId,Pid,Level,Code,Line,TimeKey, Message )
VALUES ( 'zmrecord.sh', 'zmrecord.sh', NULL, $$, $CODEVAL, \"$CODE\", 0, $TIMESTAMP, \"$MESSAGE\" );"
echo $SQL | mysql -NBqr zm
}
function log_inf() {
zm_log "INF" "2" "$@"
}
function log_err() {
zm_log "ERR" "-2" "$@"
}
# Run the ffmpeg process and keep stats file up to date while it runs - pass ffmpeg errors to zm log
function run_ffmpeg() {
MONITOR_ID="$1"
CAM_URL="$2"
OUTPUT_STATS_FN="$3"
RECTIME="$4"
OUTPUT_FN="$5"
rm "$OUTPUT_STATS_FN".errorpipe 2>/dev/null
mkfifo "$OUTPUT_STATS_FN".errorpipe
rm "$OUTPUT_STATS_FN".statpipe 2>/dev/null
mkfifo "$OUTPUT_STATS_FN".statpipe
cat "$OUTPUT_STATS_FN".errorpipe | \
while read ERR; do
log_err "Monitor: $MONITOR_ID FFMPEG: $ERR"
done &
PID_ERR_READ=$!
cat "$OUTPUT_STATS_FN".statpipe | \
while read LN; do
PARM="`echo $LN | cut -d '=' -f1`"
VAL="`echo $LN | cut -d '=' -f2`"
if [ "$PARM" = "fps" ]; then FFMPEG_FPS=$VAL; fi
if [ "$PARM" = "bitrate" ]; then FFMPEG_BITRATE=$VAL; fi
if [ "$PARM" = "frame" ]; then FFMPEG_FRAME=$VAL; fi
echo "`date +%s` $FFMPEG_FPS $FFMPEG_BITRATE $FFMPEG_FRAME" > "$OUTPUT_STATS_FN"
done &
PID_STAT_READ=$!
ffmpeg -y -loglevel error -i "$CAM_URL" \
-c:v copy -f segment \
-c:a aac \
-segment_time $RECTIME -segment_atclocktime 1 \
-segment_format_options movflags=empty_moov \
-strftime 1 "$OUTPUT_FN" \
-progress - -nostats 2>"$OUTPUT_STATS_FN".errorpipe 1>"$OUTPUT_STATS_FN".statpipe &
PID_FFMPEG=$!
# Wait for exit signal or child pid death
RUNNING=1
trap 'RUNNING=0' EXIT
trap 'RUNNING=0' INT
trap 'RUNNING=0' TERM
while [ "$RUNNING" = "1" ]; do
if ! kill -0 "$PID_FFMPEG" 2>/dev/null; then RUNNING=0; fi
if ! kill -0 "$PID_ERR_READ" 2>/dev/null; then RUNNING=0; fi
if ! kill -0 "$PID_STAT_READ" 2>/dev/null; then RUNNING=0; fi
sleep 1
done
# Shutdown the subprocess - we want to make really really sure that FFMPEG has indeed died before leaving!
# its not 100% perfect and the output from netcams can really freeze it occassionaly to the point SIGINT
# does not work and the subprocess exiting will leave a zombie ffmpeg behind
# if it does not shutdown in 4 seconds force kill it
kill $PID_FFMPEG $PID_ERR_READ $PID_STAT_READ 2>/dev/null
KILL_TIMEOUT=0
RUNNING=1
trap 'RUNNING=1' EXIT
trap 'RUNNING=1' INT
trap 'RUNNING=1' TERM
while [ "$RUNNING" = "1" ]; do
KILL_TIMEOUT=$((KILL_TIMEOUT+1))
if ! kill -0 "$PID_FFMPEG" 2>/dev/null; then RUNNING=0; fi
if [ "$KILL_TIMEOUT" -gt 4 ]; then
kill -9 $PID_FFMPEG 2>/dev/null
fi
sleep 1
done
rm "$OUTPUT_STATS_FN".errorpipe 2>/dev/null
rm "$OUTPUT_STATS_FN".statpipe 2>/dev/null
log_err Monitor: $MONITOR_ID FFMPEG Has Terminated
}
# Scan the tmp folder for <timestamp>.mp4 files
# Move them into the correct tree and create a DB entry ensuring that on premature script termination what data was
# captured so far is in place
# leave an eventid and symbolic link in the tmp folder
# the eventid keeps the info for this script to poll the video in its new location
# the symbolic link allows ffmpeg to re-open the file. FFMPEG does this when quit earlier than expected to add the moov atom (faststart)
function addIncompleteVideosToDB() {
TMPFOLDER="$1"
MONITOR_ID="$2"
STORAGEID="$3"
RECORD_TIME="$4"
WIDTH="$5"
HEIGHT="$6"
STORE_PATH="$7"
EVENT_PREFIX="$8"
# iterate all new mp4 files found
for FILEPATH in "$TMPFOLDER"/*.mp4; do
[ -f "$FILEPATH" ] || continue
# get beginning timestamp from ffmpeg created filename
FILENAME=${FILEPATH##*/}
TIMESTAMP=${FILENAME%%.mp4}
# create new db entry for video segment
SQL=" INSERT INTO Events
( MonitorId,StorageId,Name,Cause,StartDateTime,EndDateTime,Width,Height,Length,Frames,
Videoed,DiskSpace,Scheme,StateId,AlarmFrames,SaveJPEGs,Notes)
VALUES
( $MONITOR_ID,\"$STORAGEID\",\"New Event\",\"Continuous\",FROM_UNIXTIME($TIMESTAMP),
NULL,$WIDTH,$HEIGHT,1,1,1,1,\"Medium\",1,0,0,\"Injected from ffmpeg capture\");
SET @last_id = LAST_INSERT_ID();
UPDATE Events SET Name=CONCAT(\"$EVENT_PREFIX\",@last_id),DefaultVideo=CONCAT(@last_id,\"-video.mp4\") where Id=@last_id;
SELECT @last_id;"
THIS_ID=`echo $SQL | mysql -NBqr zm`
# if mysql returned a new eventid then move the video into the correct place
# ffmpeg will still retain its handle to the file and continue writing
# leave an eventid reference to the video in the tmp folder - this will allow
# polling of the file until it needs completing
if ! [ "$THIS_ID" -le 0 ]; then
DATE_PATH=$STORE_PATH/`date -d @$TIMESTAMP --rfc-3339=date`
THIS_PATH=$DATE_PATH/$THIS_ID
NEW_VIDEO_PATH="$THIS_PATH"/$THIS_ID-video.mp4
DESCFILE="$TMPFOLDER"/"$TIMESTAMP".eventid
mkdir -p $THIS_PATH
mv -f "$FILEPATH" "$NEW_VIDEO_PATH"
chown www-data:www-data "$NEW_VIDEO_PATH" "$THIS_PATH" "$DATE_PATH"
echo "$THIS_PATH/$THIS_ID-video.mp4" > "$DESCFILE"
log_inf "Monitor: $MONITOR_ID - New segment : $THIS_PATH/$THIS_ID-video.mp4 Event $THIS_ID"
fi
done
}
# Scan the tmp folder for <timestamp>.eventid files
# By checking which ones point to an mp4 that is not open ( by ffmpeg ) we can determine
# the segment is complete and close it. This can also pick up stray recordings after say a PC crash
function addCompleteVideosToDB() {
TMPFOLDER="$1"
MONITOR_ID="$2"
STORAGEID="$3"
RECORD_TIME="$4"
WIDTH="$5"
HEIGHT="$6"
STORE_PATH="$7"
# iterate all eventid files in the tmp folder
for EVENTID_FILEPATH in "$TMPFOLDER"/*.eventid; do
[ -f "$EVENTID_FILEPATH" ] || continue
EVENTID_FILENAME=${EVENTID_FILEPATH##*/}
TIMESTAMP=${EVENTID_FILENAME%%.eventid}
[[ $TIMESTAMP =~ ^[0-9]+$ ]] || continue
VIDEO_PATH="`cat "$EVENTID_FILEPATH"`"
VIDEO_FILENAME=${VIDEO_PATH##*/}
EVENTID=${VIDEO_FILENAME%%-video.mp4}
# if the video is not longer open for writing remove the eventid file, this will be the last update
LASTUPDATE=0
if [ "`fuser "$VIDEO_PATH" 2>/dev/null`" = "" ]; then
log_inf "Monitor: $MONITOR_ID Event : $EVENTID - Segment complete : $VIDEO_PATH"
rm "$EVENTID_FILEPATH"
LASTUPDATE=1
fi
# get video stats and validate them
# only spit out actual errors if this is the final update (video finished), an in-writing video
# is likley to not be stat ready early in its life
FRAMES=`mediainfo --Output="Video;%FrameCount%" "$VIDEO_PATH"`
if ! [[ $FRAMES =~ ^[0-9]+$ ]]; then
if [ $LASTUPDATE -eq 1 ]; then
log_err "Monitor: $MONITOR_ID Event : $EVENTID ERROR: mediainfo --Output=Video;%FrameCount% $VIDEO_PATH returned '$FRAMES', probable corrupt video"
fi
FRAMES=1
fi
FILESIZE=`stat -c%s "$VIDEO_PATH"`
if ! [[ $FILESIZE =~ ^[0-9]+$ ]]; then
if [ $LASTUPDATE -eq 1 ]; then
log_err "Monitor: $MONITOR_ID Event : $EVENTID ERROR: stat -c%s $VIDEO_PATH returned '$FILESIZE', integer expecected"
fi
FILESIZE=1
fi
END_TIMESTAMP=`stat -c %Y "$VIDEO_PATH"`
if ! [[ $END_TIMESTAMP =~ ^[0-9]+$ ]]; then
if [ $LASTUPDATE -eq 1 ]; then
log_err "Monitor: $MONITOR_ID Event : $EVENTID ERROR: stat -c %Y $VIDEO_PATH returned '$END_TIMESTAMP', integer expecected"
fi
LENGTH=RECORD_TIME
END_TIMESTAMP=$((TIMESTAMP+1))
fi
# update the db with stats
SQL=" UPDATE Events SET
Frames=$FRAMES,
EndDateTime=FROM_UNIXTIME($END_TIMESTAMP),
DiskSpace=$FILESIZE
WHERE Id=$EVENTID;
UPDATE Events SET Length=UNIX_TIMESTAMP(EndDateTime)-UNIX_TIMESTAMP(StartDateTime)
WHERE StartDateTime IS NOT NULL AND EndDateTime IS NOT NULL AND Id=$EVENTID;"
echo $SQL | mysql -NBqr zm
done
}
# For Bandwidth Calc
LAST_FPS_TIME=0
LAST_FILESIZE=0
LAST_BW=0
# Keep the monitor status up to date so fps etc can be viewed in the zm console
function updateMonitorStatus() {
MONITOR_ID="$1"
STATS_FILE="$2"
SHUTTINGDOWN="$3"
STATUS="NotRunning"
FPS=0
BW=0
if ! [ "$SHUTTINGDOWN" = "1" ]; then
if [ -f "$STATS_FILE" ]; then
IFS=' ' read -r TS FPS BW FRAME <<< `cat "$STATS_FILE"`
if [ -z "${TS//[0-9]}" ] && [ -n "$TS" ]; then
AGE=$(($(date +%s)-TS))
else
AGE=1000
fi
if [ "$AGE" -lt 4 ]; then
STATUS="Connected"
fi
fi
fi
# Calc CaptureBandwith in the same way as ZM in zm_monitor.cpp:
# unsigned int new_capture_bandwidth = (new_camera_bytes - last_camera_bytes)/(now-last_fps_time);
# Unfortunately, BASH can't trap a divide by zero so we have to check
# Get Filesize for Bandwidth calc. Ffmpeg buffers quite a lot so see if the file has changed
# size before calculating the Bandwidth.
FILESIZE=`stat -c%s "$NEW_VIDEO_PATH" 2>/dev/null`
if [ ${FILESIZE:-0} -gt ${LAST_FILESIZE:-0} ]
then
ELAPSED_SECONDS=$(($TS - $LAST_FPS_TIME))
if [ $ELAPSED_SECONDS > 0 ]
then
FILESIZE_INCREASE=$(( ${FILESIZE:-0} - ${LAST_FILESIZE:-0} ))
LAST_BW=$(( $FILESIZE_INCREASE / $ELAPSED_SECONDS ))
LAST_FPS_TIME=$TS
LAST_FILESIZE=$FILESIZE
fi
fi
if [ "$FPS" == "" ]; then
FPS=0
fi
SQL=" REPLACE INTO Monitor_Status (MonitorId, Status, CaptureFPS, CaptureBandwidth)
VALUES
('$MONITOR_ID','$STATUS', '$FPS', ${LAST_BW:-0})"
echo $SQL | mysql -NBqr zm
}
# Main function -
function zmRecord() {
MONITOR_ID=$1
# Read Monitor Info From DB
IFS=$'\t'
read -r WIDTH HEIGHT READPATH STORAGEID EVENT_PREFIX <<< `mysql -NBqr zm -e "select Width,Height,Path,StorageId,EventPrefix from Monitors where Id=$MONITOR_ID"`
read -r STORAGE <<< `mysql -NBqr zm -e "select Path from Storage where Id=$STORAGEID"`
# Hmmmm.... on a fresh install of ZM 1.34, the Monitor has a StorageId of 0 but the Storage table
# starts at Id 1. Handle a blank Storage Path.
if [ -z "$STORAGE" -a $STORAGEID -eq 0 ]
then
read -r STORAGE <<< `mysql -NBqr zm -e "select Path from Storage where Id=1"`
fi
STORE_PATH=$STORAGE/$MONITOR_ID
if [ -z "$READPATH" -o -z "$STORE_PATH" -o -z "$STORAGE" ]; then
log_err "$0 Monitor Data Error. Got: Width $WIDTH Height $HEIGHT ReadPath $READPATH StorePath $STORE_PATH"
exit 1
fi
# create the working folders & files
# stats txt is constantly updated and placed in ram ( /dev/shm )
TMPFOLDER="$STORE_PATH/../$MONITOR_ID-zmrecordtmp"
FFMPEG_STAT_OUTPUT_FILE=/dev/shm/zmrecord-m$MONITOR_ID-stats.txt
FFMPEG_OUTPUT_FILENAME="$TMPFOLDER/%s.mp4"
mkdir -p "$TMPFOLDER"
rm "$FFMPEG_STAT_OUTPUT_FILE" 2>/dev/null
touch "$FFMPEG_STAT_OUTPUT_FILE"
chmod a+r "$FFMPEG_STAT_OUTPUT_FILE"
# Main loop exit trap triggered by EXIT / INT / TERM signals
RUNNING=1
trap 'RUNNING=0' EXIT
trap 'RUNNING=0' INT
trap 'RUNNING=0' TERM
# Main loop - start the ffmpeg processes and monitor
# regularly run updates to check ffmpegs video and stat output and transfer
# to zm database / file tree
log_inf "Monitor: $MONITOR_ID Starting zmrecord.sh"
FFMPEG_PID=-100
while [ "$RUNNING" = "1" ]; do
# testing this script one time my tmp folder vanished ( zm cleanup? )
# and this script was unable to recover so I added this
mkdir -p "$TMPFOLDER"
# start subprocesses / restart them if they stopped
if ! kill -0 "$FFMPEG_PID" 2>/dev/null; then
log_inf "Monitor: $MONITOR_ID Starting FFMPEG process"
run_ffmpeg "$MONITOR_ID" "$READPATH" "$FFMPEG_STAT_OUTPUT_FILE" "$RECORD_TIME" "$FFMPEG_OUTPUT_FILENAME" &
FFMPEG_PID=$!
fi
# check ffmpeg is actually working by monitoring the frame count
FFMPEG_STATS_FRAMES=`cat "$FFMPEG_STAT_OUTPUT_FILE" | cut -d " " -f4`
if [ "$PREV_FFMPEG_STATS_FRAMES" = "$FFMPEG_STATS_FRAMES" ]; then
if [ "$PREV_FFMPEG_STATS_FRAMES_SAMECOUNT" = "" ]; then
PREV_FFMPEG_STATS_FRAMES_SAMECOUNT=0
fi
PREV_FFMPEG_STATS_FRAMES_SAMECOUNT=$((PREV_FFMPEG_STATS_FRAMES_SAMECOUNT+1))
else
PREV_FFMPEG_STATS_FRAMES_SAMECOUNT=0
fi
PREV_FFMPEG_STATS_FRAMES="$FFMPEG_STATS_FRAMES"
if [ "$PREV_FFMPEG_STATS_FRAMES_SAMECOUNT" -gt 5 ]; then
if kill -0 "$FFMPEG_PID" 2>/dev/null; then
log_err "Monitor: $MONITOR_ID FFMPEG has stalled - killing"
kill $FFMPEG_PID
PREV_FFMPEG_STATS_FRAMES_SAMECOUNT=0
fi
fi
# run the video / db update polling functions
addCompleteVideosToDB "$TMPFOLDER" "$MONITOR_ID" "$STORAGEID" "$RECORD_TIME" "$WIDTH" "$HEIGHT" "$STORE_PATH" "$EVENT_PREFIX"
sleep 4
updateMonitorStatus "$MONITOR_ID" "$FFMPEG_STAT_OUTPUT_FILE" 0
addIncompleteVideosToDB "$TMPFOLDER" "$MONITOR_ID" "$STORAGEID" "$RECORD_TIME" "$WIDTH" "$HEIGHT" "$STORE_PATH" "$EVENT_PREFIX"
done
# Finish up
if ! kill -0 "$FFMPEG_PID" 2>/dev/null; then
kill "$FFMPEG_PID" 2>/dev/null
fi
addCompleteVideosToDB "$TMPFOLDER" "$MONITOR_ID" "$STORAGEID" "$RECORD_TIME" "$WIDTH" "$HEIGHT" "$STORE_PATH" "$EVENT_PREFIX"
updateMonitorStatus "$MONITOR_ID" "$FFMPEG_STAT_OUTPUT_FILE" 1
log_inf "Monitor: $MONITOR_ID Ended zmrecord.sh"
}
if [ $# -ne 1 ]; then
echo "Syntax: $0 <Monitor Id>"
exit 1
fi
MONITOR_ID=$1
zmRecord $MONITOR_ID
#
# Fire up zmrecord for multiple cameras.
#
# Enable with:
#
# systemctl enable zmrecord@<MonitorId>.service
#
# the recording should start
#
# Start each recording process automatically at boot with:
#
# systemctl start zmrecord@<MonitorId>.service
#
#
[Unit]
Description=Record Stream from Camera %I
After=zoneminder.service
[Service]
Type=simple
ExecStart=/opt/zmrecord.sh %I
Restart=on-failure
[Install]
WantedBy=multi-user.target
@ruffle-b
Copy link

Hi Matt,

Thanks for making those improvements - very nice.

I've just upgrade my ZM to 1.34 and revisited this, noticed your improvements and a request to honour EventPrefix on the forum so I forked your version, made the changes for EventPrefix, added some Bandwidth calcs just to keep the ZM console looking sane and handled a curious 1.34 issue on Storage Areas Ids.

Anyway, my version (0.7) is on the forked copy: https://gist.github.com/ruffle-b/5ab39b9b6f456d694f7665678698bf5f

I don't think Gist supports formal pulls but feel free to merge my changes and call your copy the master... or I can leave my fork live; not bothered which.

@mwvent
Copy link
Author

mwvent commented Feb 20, 2020

Thank you - I have merged the changes into this one. I dont think it will hurt to keep your fork live as I do not go on github ( or the PC ) much when all the stuff is working so well 💯
Again thanks for putting this all forward - the Zoneminder server is purring along at about 40% CPU with 3 HD streams managed by this script and their respective sub-streams being motion analysed traditionally with zoneminder. It was completley saturated before! Looking foward to even more CPU savings to come by using the cameras own motion detection to trigger Zoneminder.
I am now in a position to replace this old high wattage CPU ( Q6600 ) with a nice quiet low wattage APU which is kind of where we all want to be with a 24/7 running PC :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment