Skip to content

Instantly share code, notes, and snippets.

@BipedalTV
Forked from dmoo1790/gist:507582
Last active August 21, 2020 23:36
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save BipedalTV/3174241 to your computer and use it in GitHub Desktop.
Save BipedalTV/3174241 to your computer and use it in GitHub Desktop.
Script to cut MythTV recordings 0.26+fixes
#!/bin/bash
# MythCutKey Version 0.15e
# This script cuts myth recordings at key frames using the MythTV seek table.
# Why? No external programs required, lossless and fast.
# Updates the myth database with sql calls.
# Rebuilds output file seek table with mythutil (mythcommflag at present).
# Output files may NOT be easily editable again and may have glitches at cut points.
#
# Undo option allows restoration of original recording and associated database
# records including seek table and markup table.
# NOTE: The undo function requires a directory for which mysql has write permissions.
# The default is /tmp/ however you will lose your undo information when you reboot.
# Undo file names are the recording file base name with seekN.sql, markN.sql or recN.sql
# appended, where N is a number starting from zero and the highest number indicating the
# latest undo files. The undo files are not huge but may eat up disk space over time
# after many edits.
#
# The script does not delete the original recording so you need disk free space equal
# to something less than the original recording file size depending on how much you cut
# from the file.
# MythTV versions: tested 0.26 - 0.28 probably okay with 0.24+fixes, 0.25+fixes
# just comment out mythutil --notification line for pre 0.27.
#
# changelog
# 0.12 : tidied some sql .
# 0.13 : fixed the "dd" byte calcs, tested/checked with spreadsheet using real & synthesised data..
# : confirmed all cut points in final file +/-1 keyframe from expected.
# 0.14 : mysql db login from mysql.txt
# 0.15 : db creds from config.xml, -f file option works
# : 0.26+fixes can not play cut & concated TS files without patch Ticket #11159
# : mysql outfile method stopped working..
# : creates tmp sub folder..
# c : keep first 3 pkts of file PSI/PAT/PMT & SPS if lucky
# d : notification message..mythutil --message & --notification require IP address
# e : was still using obsolete config.xml entries https://github.com/MythTV/mythtv/commit/2e1820c4662531f25e52f02f1b309c6a548b71ec
#
OVERWRITE=0
OUTPUT_DIR=""
CHANID=-9999
STARTTIME=""
rec_dir=""
basename=""
UNDO=0
# changed mysql outfile to echo redirect to file because it stopped working ..
# just set UNDO_DIR chown mythtv:mythtv {old: chgrp mysql" the dir and "chmod ug+rwX"}
UNDO_DIR="/tmp/MythCutKey/"
STARTSECS=`date +%s`
# recordedseek (chanid starttime mark offset type)
MARK_CUT_END=0
MARK_CUT_START=1
MARKTYPES="$MARK_CUT_END,$MARK_CUT_START"
# MARK_GOP_BYFRAME = 9
MARKKEYTYPES="6,7,9"
# recordedmarkup (chanid starttime mark type data) only ?
MARK_ASPECT_CUSTOM=14
MARK_VIDEO_WIDTH=30
MARK_VIDEO_HEIGHT=31
MARK_VIDEO_RATE=32
MARK_DURATION_MS=33
MARK_TOTAL_FRAMES=34
USAGE="$0 -c channel_id -s starttimeutc [-d dir] [-f filename] [-o directory] [-h] [-Z]
[-u undo_dir] [-U]
-h Help/usage
-c Channel ID (required, %CHANID% from Mythtv)
-s Start time (required, %STARTTIMEUTC% from Mythtv)
-d Directory where the recording is store (optional, %DIR% from Mythtv)
-f Recording file name (optional, %FILE% from Mythtv)
-o Output directory (optional, recording directory if not specified)
-Z Overwrite original recording (optional, default is false, ignored if -o specified)
-u Directory where undo info is saved (optional, mysql must have permission to write
to this directory, default is /tmp/)
-U Undo the last edit
Example usage overwriting original recording (original is renamed, not deleted)
mythcutkey -c %CHANID% -s %STARTTIMEUTC% -Z
Example usage to save the file to a different directory
mythcutkey -c %CHANID% -s %STARTTIMEUTC% -o /your/directory/
Example usage specifying an undo directory
mythcutkey -c %CHANID% -s %STARTTIMEUTC% -Z -u /myundodir
Example usage to undo the last edit
mythcutkey -c %CHANID% -s %STARTTIME% -U -u /myundodir
Warning: The end files may not be editable and/or playable outside MythTV. Keep the original
recording if you think you may want to do further edits or conversions in future. This script does
NOT delete the original recording. It is renamed with the extension \".old\"."
while getopts "c:s:o:d:f:t:u:hZU" opt
do
case $opt in
c )
CHANID="$OPTARG"
;;
s )
STARTTIME="$OPTARG"
;;
d )
rec_dir="$OPTARG"
;;
f )
basename="$OPTARG"
;;
h )
echo "Usage: $USAGE" >&2
exit 0
;;
o )
OUTPUT_DIR="$OPTARG"
;;
u )
UNDO_DIR="$OPTARG"
;;
Z ) OVERWRITE=1 # Will be reset to false if -o directory is specified
;;
U ) UNDO=1
;;
? )
echo "Invalid option: -$OPTARG" >&2
echo "Usage: $USAGE" >&2
exit -1
;;
esac
done
if [ "$OUTPUT_DIR" != "" ]; then
OVERWRITE=0
fi
if [ "$basename" == "" ]; then
if [ $CHANID == -9999 -o "$STARTTIME" == "" ]; then
echo "Filename or Channel ID and Start Time missing" >&2
echo "Usage: $USAGE" >&2
exit -1
fi
fi
if [ ! -d $UNDO_DIR ]; then
mkdir $UNDO_DIR
fi
if [ ! -d $UNDO_DIR ]; then
echo "Can't find/create undo directory "$UNDO_DIR", aborting" >&2
exit -1
fi
#########################################
# db login cred
#########################################
MYTHCFG=""
# mythbackend in Fedora packages has $HOME=/etc/mythtv
if [ -e $HOME/config.xml ]; then
MYTHCFG="$HOME/config.xml"
fi
if [ -e $HOME/.mythtv/config.xml ]; then
MYTHCFG="$HOME/.mythtv/config.xml"
fi
if [ "$MYTHCFG" = "" ]; then
echo >>$LOGFILE "No config.xml found in $HOME or $HOME/.mythtv - exiting!"
exit -1
fi
# DB username and password
MYTHHOST=`sed -n 's|<Host>\(.*\)</Host>|\1|p' $MYTHCFG | sed 's/^[ \t]*//;s/[ \t]*$//'`
MYTHUSER=`sed -n 's|<UserName>\(.*\)</UserName>|\1|p' $MYTHCFG | sed 's/^[ \t]*//;s/[ \t]*$//'`
MYTHPASS=`sed -n 's|<Password>\(.*\)</Password>|\1|p' $MYTHCFG | sed 's/^[ \t]*//;s/[ \t]*$//'`
MYTHDB=`sed -n 's|<DatabaseName>\(.*\)</DatabaseName>|\1|p' $MYTHCFG | sed 's/^[ \t]*//;s/[ \t]*$//'`
# echo "[$MYTHHOST] [$MYTHUSER] [$MYTHPASS] [$MYTHDB]"
USERPASSTABLE="-h$MYTHHOST -u$MYTHUSER -p$MYTHPASS $MYTHDB"
#echo $USERPASSTABLE
#############################################################################
# Find where the recording is stored if not specified with -d and -f options
#############################################################################
if [ "$rec_dir" == "" ]; then
storage_dirs=`echo "select dirname from storagegroup;" | mysql -N $USERPASSTABLE `
if [ "$basename" == "" ]; then
basename=`echo "select basename from recorded where chanid=$CHANID and starttime=\"$STARTTIME\";" \
| mysql -N $USERPASSTABLE `
else
CHANID=`echo "select chanid from recorded where basename=\"$basename\";" | mysql -N $USERPASSTABLE `
STARTTIME=`echo "select starttime from recorded where basename=\"$basename\";" | mysql -N $USERPASSTABLE `
fi
#echo $storage_dirs
found=0
for f in $storage_dirs; do
if [ -e "$f$basename" ]; then
rec_dir=$f
found=1
fi
done
echo $rec_dir$basename" | "$CHANID" | "$STARTTIME
if [ $found == 0 ]; then
echo "Can't find recording, aborting" >&2
exit -1
fi
fi
# Set output directory to recording directory if not specified with -o option
if [ "$OUTPUT_DIR" == "" ]; then
OUTPUT_DIR=$rec_dir
fi
#############################################################################
# Undo the last edit if -U option specified
#############################################################################
if [ $UNDO == 1 ]; then
if [ ! -e $rec_dir$basename".old" ]; then
echo "Can't find original recording "$rec_dir$basename".old, aborting" >&2
exit -1
fi
# Don't delete. Just rename.
mv $rec_dir$basename $rec_dir$basename".new"
mv $rec_dir$basename".old" $rec_dir$basename
# Undo various database changes
num=0
while [ -e $UNDO_DIR$basename.seek$num.sql ]; do
let num++
done
let num--
echo "delete from recordedseek where chanid=$CHANID and starttime=\"$STARTTIME\";" | mysql $USERPASSTABLE
echo "load data infile \"$UNDO_DIR$basename.seek$num.sql\" into table recordedseek;" | mysql $USERPASSTABLE
echo "delete from recordedmarkup where chanid=$CHANID and starttime=\"$STARTTIME\";" | mysql $USERPASSTABLE
echo "load data infile \"$UNDO_DIR$basename.mark$num.sql\" into table recordedmarkup;" | mysql $USERPASSTABLE
echo "delete from recorded where chanid=$CHANID and starttime=\"$STARTTIME\";" | mysql $USERPASSTABLE
echo "load data infile \"$UNDO_DIR$basename.rec$num.sql\" into table recorded;" | mysql $USERPASSTABLE
exit 0
fi
#############################################################################
# Edit the recording based on cuts at keyframes
#############################################################################
thefile=$rec_dir$basename
outfile=$OUTPUT_DIR$basename".new"
# Get the cuts from recordedmarkup table and store in an array
# returns zero indexed one dim array, 2 elements per seektable entry
# always returns [type, mark] (2 elements) as cuts[0] & cuts[1] if not blank..
cuts=( `echo "select type,mark from recordedmarkup where chanid=$CHANID and starttime=\"$STARTTIME\" \
and type in ($MARKTYPES) order by mark;" | mysql -N $USERPASSTABLE ` )
echo $cuts
pieces=${#cuts[@]}
if [ $pieces == 0 ]; then
echo "No cuts in seektable, aborting" >&2
exit -1
fi
# Insert a cut mark record for the start of the recording
let k=pieces-1
while [ $k -gt 0 ]; do
cuts[k+2]=${cuts[k]}
cuts[k+1]=${cuts[k-1]}
let k-=2
done
let pieces=pieces+2
# Cut'(1)= $MARK_CUT_END if first cut is not the start of recording
cuts[1]=1
if [ ${cuts[2]} -ne $MARK_CUT_END ]; then
cuts[0]=$MARK_CUT_END
else
cuts[0]=$MARK_CUT_START
fi
echo ${cuts[0]} ${cuts[1]} ${cuts[2]} ${cuts[3]}
# Check if last cut is not end of recording, i.e., we want to keep the end of the recording
if [ ${cuts[pieces-2]} == $MARK_CUT_END ]; then
lastcut=${cuts[pieces-1]}
lastseek=""
lastseek=`echo "select mark from recordedseek where chanid=$CHANID and starttime=\"$STARTTIME\" order by -mark LIMIT 1;" | \
mysql -N $USERPASSTABLE | tail -n 1 `
echo "lastcut = "$lastcut" lastseek = "$lastseek
if [ "$lastseek" != "" ]; then
if [ $lastseek -gt $lastcut ]; then
cuts[pieces]=$MARK_CUT_START
let cuts[pieces+1]=$lastseek # Assume end of recording is within 100 frames of last seek table entry
let pieces=pieces+2
fi
fi
fi
offset_start=( `echo "select min(offset) from recordedseek where chanid=$CHANID and starttime=\"$STARTTIME\";" | mysql -N $USERPASSTABLE | tail -n 1 ` )
echo $offset_start
let pieces=pieces-1
i=0
k=0
part=1
while [ $i -le $pieces ]; do
if [ ${cuts[i]} -eq 0 ]; then # Look for a cut end, i.e., the start of a segment we want to keep
# Find keyframes before or at keep piece
cutstart=$((${cuts[i+1]}+1))
cutlimit=$(($cutstart-200))
key=( `echo "select mark,offset from recordedseek where mark>=$cutlimit and mark<=$cutstart and chanid=$CHANID and starttime=\"$STARTTIME\" \
and type in ($MARKKEYTYPES) order by -offset LIMIT 1;" | mysql -N $USERPASSTABLE | tail -n 1 ` )
cutstart=$((${key[0]}))
key1=$((${key[1]}))
if [ $cutstart -le 1 ]; then
key1=0
cutstart=0
fi
# find keyframe after keep piece
# why ? because can not calculate the offset without better seektable.
lastseek=`echo "select mark from recordedseek where chanid=$CHANID and starttime=\"$STARTTIME\" order by -mark limit 1;" | \
mysql -N $USERPASSTABLE | tail -n 1`
cutlimit=$(($lastseek))
cutend=$((${cuts[i+3]-2}))
key=( `echo "select mark,offset from recordedseek where mark>=$cutend and mark<=$cutlimit and chanid=$CHANID and starttime=\"$STARTTIME\" \
and type in ($MARKKEYTYPES) order by offset limit 1;" | mysql -N $USERPASSTABLE | tail -n 1` )
echo "mark,offset = "${key[@]}" , lastseek= "$lastseek
cutend=$((${key[0]}))
key2=$((${key[1]}))
#key1=$(($key1+$offset_start))
#key2=$(($key2+$offset_start))
# Check if cutend is past the last record in the seek table i.e. blank..
if [ $cutend -eq 0 ]; then
key2=`echo "select filesize from recorded where chanid=$CHANID and starttime=\"$STARTTIME\";" | \
mysql -N $USERPASSTABLE | tail -n 1 `
let cutend=$lastseek
fi
bytes=$(($key2-$key1))
let k++
# keep first 3 pkts (188bytes = 564B) by adjusting key1 if close to first cut.
# need to do before block calculations
if [ $key1 -lt 564 ]; then
key1=0
fi
echo "Segment start="$key1" bytes (frame="$cutstart", cutpoint frame was "${cuts[i+1]}")"
echo "Segment end="$key2" bytes (frame="$cutend", cutpoint frame was "${cuts[i+3]}")"
kblocks=188 # efficient as pkt size
blocksize=$((1024*$kblocks))
blockstart=$(((($key1+$blocksize+1)/$blocksize)*$blocksize))
blockstartlimited=$key2
pre=0 # lowest block aligned offset
if [ $blockstart -lt $key2 ]; then
pre=$(($blockstart))
blockstartlimited=$(($blockstart-1))
# pre used as relative offset (count) not abs offset
pre=$(($pre-$key1))
fi
blocks=$((($key2-($key2/$blocksize)-$key1-($key1/$blocksize)-1)/$blocksize))
# highest block aligned offset
poststart=$((($key2/$blocksize)*$blocksize))
post=$(($key2-$poststart))
if [ $poststart -lt 1 ]; then
post=0
fi
echo "blockstart="$blockstart" pre="$pre" blocks="$blocks" poststart="$poststart" post="$post
echo "Total bytes in segment="$bytes" ="$pre" bytes + "$(($blocks*$blocksize))" blocks(byte) + "$post" bytes"
# Cut and paste append the segment with dd
echo "Cutting and pasting segment "$part"a"
if [ $part == 1 ]; then
if [ $key1 -gt 0 ]; then # first keep piece not near start
ionice -c3 dd ibs=188c obs=188c skip=0 count=3 if=$thefile of=$outfile
ionice -c3 dd ibs=1c obs=1c skip=$key1 count=$pre if=$thefile of=$outfile oflag=append conv=notrunc
else
if [ $part == 1 ]; then
ionice -c3 dd ibs=1c obs=1c skip=$key1 count=$pre if=$thefile of=$outfile
fi
fi
else
ionice -c3 dd ibs=1c obs=1c skip=$key1 count=$pre if=$thefile of=$outfile oflag=append conv=notrunc
fi
if [ $blocks -gt 0 ];then
echo "Cutting and pasting segment "$part"b"
ionice -c3 dd bs=$kblocks"K" skip=$(($blockstart/$blocksize)) count=$blocks if=$thefile of=$outfile oflag=append conv=notrunc
fi
if [ $post -gt 0 ]; then
echo "Cutting and pasting segment "$part"c"
ionice -c3 dd ibs=1c obs=1c skip=$poststart count=$post if=$thefile of=$outfile oflag=append conv=notrunc
fi
let part++
# Increment loop counter by 4 since array has cut start/end pairs plus cut types
let i+=4
else
# Skips the case where the first cut point is a cut start, i.e., when start of recording is deleted
let i+=2
fi
done
segs=$k
#############################################################################
# Reset the cut list, seek table, etc. for final file
#############################################################################
finalfile=$OUTPUT_DIR$basename
if [ $OVERWRITE == 1 ]; then
# Don't delete original. Just rename.
mv $rec_dir$basename $rec_dir$basename".old"
mv $outfile $finalfile
# Delete tmp file if it exists
if [ -e $OUTPUT_DIR$basename".tmp" ]; then
ionice -c3 rm $OUTPUT_DIR$basename".tmp"
fi
# Backup various data so we can undo the changes
num=0
while [ -e $UNDO_DIR$basename.seek$num.sql ]; do
let num++
done
# outfile not work anymore..with GRANT FILE *.* etc & all folder permissions.
echo "select * from recordedseek where chanid=$CHANID and starttime=\"$STARTTIME\";" | mysql $USERPASSTABLE \
> $UNDO_DIR$basename.seek$num.sql
echo "select * from recordedmarkup where chanid=$CHANID and starttime=\"$STARTTIME\";" | mysql $USERPASSTABLE \
> $UNDO_DIR$basename.mark$num.sql
echo "select * from recorded where chanid=$CHANID and starttime=\"$STARTTIME\";" | mysql $USERPASSTABLE \
> $UNDO_DIR$basename.rec$num.sql
# new start time format has space+chars "YYYY-MM-DD HH:MM:SS" ..& can't pass it to mythcommflag or util
temp1=${STARTTIME//:}
temp2=${temp1//-}
temp3=${temp2// } # YYYYMMDDHHMMSS
# echo "ionice -c3 mythcommflag --rebuild --chanid $CHANID --starttime $temp3"
# ionice -c3 mythcommflag --rebuild --file $rec_dir$basename
ionice -c3 mythcommflag --rebuild --chanid $CHANID --starttime $temp3
mythutil --clearcutlist --chanid $CHANID --starttime $temp3
mythutil --clearskiplist --chanid $CHANID --starttime $temp3
else
# Don't accidentally overwrite an existing file
if [ -e $finalfile ]; then
i=1
while [ -e $finalfile"."$i".mpg" ]; do
let i++
done
mv $outfile $finalfile"."$i".mpg"
else
mv $outfile $finalfile
fi
fi
ENDSECS=`date +%s`
SECS=$(($ENDSECS-$STARTSECS))
MINS=$(($SECS/60))
SECS=$(($SECS-$MINS*60))
echo "Elapsed time: "$MINS" minutes "$SECS" seconds"
mythutil --bcastaddr 192.168.1.32 --notification --type normal --message_text "Finished Job: "$basename --description "MythCutKey mpegts cutter" --timeout=3
@BipedalTV
Copy link
Author

Note that mpeg-ts has audio - video pkt offset of up to 2-3 seconds. There will be missing audio data at the start of a "keep piece".

@BipedalTV
Copy link
Author

Need to set the correct FE IP address to get notification messages.

@grandmastermarclar
Copy link

The config.xml in MythTV 0.28 uses Host, UserName, and Password. Lines 166, 167, and 168 need to be amended accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment