Composing two recorded OpenVidu streams into one

README.md

This covers the usecase when 1:1 OpenVidu session recorded in INDIVIDUAL mode, participants may join and leave at different moments, but you only want to keep the portion where both are present. This command produces video file with two recorded streams side-by-side, and both audio tracks merged (broken down for readability):

 ffmpeg \
  -ss 1054403ms -c:a libopus -i .\str_CAM_Fs1t_con_CjOpssSpAe.webm \
  -i -c:a libopus .\str_CAM_GQcn_con_RzVjNmZY6i.webm \
  -filter_complex "[0][1]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[0s][1s];
                   [0s][1s]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[1s][0s];
                   [0s][1s]hstack=inputs=2:shortest=1,setsar=1[v];
                   [0:a][1:a]amerge,aresample=async=1000[a]" \
  -map "[v]" -map "[a]" -ac 2 -r 30 -crf 23 test.mp4

Explanation:

-ss 1054403ms - this needs to be applied to the stream that started first, to skip its beginning until the second stream starts. Insert startTimeOffset value of the file that starts later

scale2ref - scales one video relatively to the other, read more: here and here; must be two times in the filtergraph; we must scale even if metadata say that both resolutions are the same, because actual resolution of WebRTC is dynamic

hstack=shortest=2 - stop writing output when the shortest of two videos ends (default is longest)

if getting error like "width/height not divisible by 2", use [0s][1s]hstack=inputs=2:shortest=1,setsar=1[vp];[vp]pad=ceil(iw/2)*2:ceil(ih/2)*2[v];

amerge - merge two audio tracks; -ac 2 mix to stereo (2 channels) in the output, otherwise channels just add up

aresample=async=1000 - not really sure, but sometimes it seems to help prevent out-of-sync audio as well as -r 30 (contant framerate) and -crf 23

-c:a libopus to handle opus audio tracks properly, read: https://trac.ffmpeg.org/ticket/4641#comment:3

without -vsync 0 mp4 has weird 1000fps and is unplayable; thanks to https://stackoverflow.com/a/40183660/3120296

convert.sh

#!/bin/bash

# This is a very simple example script, that downloads a recording from AWS instance,
# unzips it, and combines two .webm recorded streams into one .mp4 file,
# composed in "side-by-side" layout.
#
# It uses jq JSON-parser (https://stedolan.github.io/jq/)
# 
# It only deals with two streams for a specific use case, please check the README.md in this gist.
# 
# Usage: ./convert.sh <SESSION_ID>

# 1. Download (ssh access is configured in ~/.ssh/config
scp aws:/opt/openvidu/recordings/$1/$1.zip ./

# 2. Check integrity
remote_hash=$(ssh aws md5sum /opt/openvidu/recordings/$1/$1.zip)
local_hash=$(md5sum ./$1.zip)

if [ "${local_hash%% *}" = "${remote_hash%% *}" ]; then
  echo "Integrity check OK"
  unzip ./$1.zip -d ./$1
else
  echo "Integrity check failed" && exit 1
fi


cd ./$1

# 3. Extract info from .json
count=$(jq -r '.files | length' $1.json)

if [[ $count -gt 2 ]]; then
  echo "More than 2 streams in the archive" && exit 1
fi

RSTREAM=$(jq -r '.files[0].streamId' $1.json)
ROFFSET=$(jq -r '.files[0].startTimeOffset' $1.json)
LSTREAM=$(jq -r '.files[1].streamId' $1.json)
LOFFSET=$(jq -r '.files[1].startTimeOffset' $1.json)
DELTA=$((LOFFSET - ROFFSET))

# 4. Determine which stream starts first
if [[ $DELTA -ge 0 ]]; then

  # Standard case: first stream starts earlier
  LEFT=$RSTREAM
  RIGHT=$LSTREAM

else
  LEFT=$LSTREAM
  RIGHT=$RSTREAM
fi

# Absolute value of DELTA
OFFSET=$(echo $DELTA | tr -d -)

# 5. Real magic starts here
ffmpeg -y -hide_banner -ss ${OFFSET}ms -c:a libopus -i ./${LEFT}.webm -c:a libopus -i ./${RIGHT}.webm -filter_complex "[0][1]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[0s][1s];[0s][1s]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[1s][0s];[1s][0s]hstack=inputs=2:shortest=1,setsar=1[v];[0][1]amerge,aresample=async=1000[a]" -map "[v]" -map "[a]" -r 30 -crf 23 -ac 2 ./$1.mp4

Hi!
Seems
[0s][1s]scale2ref='ohmdar':'if(lt(main_h,ih),ih,main_h)'[1s][0s];
should be
**[1s][0s]**scale2ref='ohmdar':'if(lt(main_h,ih),ih,main_h)'[1s][0s];

One more thing

To merge audio in proper way without sync problems this helps me:

[0:a]aresample=async=1000[0sync];[1:a]aresample=async=1000[1sync];[0sync][1sync]amix[a]