How to predict and fix audio desynchronization when uploading Twitch streams to YouTube.

twitch-to-youtube-desync.md

The problem

When I download streams from Twitch, I remux them into MP4 because MPEG-TS doesn't support -movflags faststart, is not supported by HTML5 players and can't be uploaded directly to YouTube. Sometimes this process changes framerate of the video stream. Incorrect FPS somehow corrupts the video during processing on YouTube, which leads to horrible audio desynchronization on any resolutions below 1080p60.

My working theory is that Twitch doesn’t fill the voids in the stream when frames are lost. This is not critical for MPEG-TS and FLV containers, however, the MP4 container tries to compute the global average FPS. YouTube somehow relies on the global FPS during video processing, which leads to a decrease in the duration of the video stream, and causes its desynchronization with audio.

Check if video is safe to upload (manual method)

Good video (60 fps):

# ffmpeg -i Streams/good.mp4

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Streams/good.mp4':
  Duration: 06:04:58.97, start: 0.000000, bitrate: 7672 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 7495 kb/s, 60 fps, 60 tbr, 90k tbn, 120 tbc (default)

Bad video (59.99 fps):

# ffmpeg -i Streams/bad.mp4

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Streams/bad.mp4':
  Duration: 05:58:20.47, start: 0.000000, bitrate: 7677 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 7499 kb/s, 59.99 fps, 60 tbr, 90k tbn, 120 tbc (default)

Check if video is safe to upload (automatic method)

function ffprobe_entry { # filename, stream, entry
    ffprobe \
        -v error \
        -select_streams "$2" \
        -show_entries "$3" \
        -of default=noprint_wrappers=1:nokey=1 \
        "$1" | tail -n1
}

function is_correct_framerate { # filename
    avg_fps=$(ffprobe_entry "$1" v:0 "stream=avg_frame_rate")
    r_fps=$(ffprobe_entry "$1" v:0 "stream=r_frame_rate")

    # Allow no more than 0.05 seconds of desynchronization per hour
    [ "$(echo "60 * ($r_fps - $avg_fps) < 0.05" | bc -l)" -eq 1 ]
}

is_correct_framerate Streams/good.mp4 && echo YES || echo NO
is_correct_framerate Streams/bad.mp4 && echo YES || echo NO

Output:

YES
NO

Fix video without re-encoding

The easiest and the fastest way is to remux the video into FLV. But first it needs to be remuxed back into MPEG-TS, otherwise ffmpeg will carry incorrect framerate into the new container.

ffmpeg -v error -i INPUT.mp4 -c copy -f mpegts - | \
    ffmpeg -y -hide_banner -i - -c copy -bsf:a aac_adtstoasc OUTPUT.flv

The resulting OUTPUT.flv should have the same framerate as the original source (60 FPS in this example).

ffmpeg -i bad.flv

Input #0, flv, from 'bad.flv':
  Duration: 05:58:20.48, start: 0.010000, bitrate: 7675 kb/s
    Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 1k tbn, 120 tbc

That means this video can now be uploaded to YouTube.

Do you know another fix?

Feel free to leave comments with your solutions. I still can't figure out how to remux MPEG-TS into MP4 without altering the framerate. I have tried MP4Box, mkvtoolnix, FFmpeg 2.8-4.2, all without success. Full re-encoding is not an option for me because I don't want to stress my hardware this much.