I have multiple mp4 video files and I want to composite them into a single video. Each stream is an mp4 video. They are of different lengths, and each file also has audio.
The tricky thing is, I want the layout to change depending on how many streams are currently visible.
As a concrete example, say I have 3 video files:
| File | Duration | Start | End |
|---|---|---|---|
| a.mp4 | 30s | 0s | 30s |
| b.mp4 | 10s | 10s | 20s |
| c.mp4 | 15s | 15s | 30s |
So at t=0 seconds, I want the video to look like this:
+-------------------------+
| |
| |
| |
| |
| a.mp4 |
| |
| |
| |
| |
| |
+-------------------------+
At t=10s, I want the video to look like this:
+------------------------------+--------+
| | |
| | |
| | a.mp4 |
| | |
| +--------+
| b.mp4 |
| |
| |
| |
| |
| |
+------------------------------+
At t=15s, I want the video to look like this:
+------------------------------+--------+
| | |
| | |
| | a.mp4 |
| | |
| +--------+
| b.mp4 | |
| | |
| | c.mp4 |
| | |
| +--------+
| |
+------------------------------+
And at t=20s until the end, I want the video to look like this:
+------------------------------+--------+
| | |
| | |
| | a.mp4 |
| | |
| +--------+
| c.mp4 |
| |
| |
| |
| |
| |
+------------------------------+
Ideally there would be some animated transitions between the states, but that's not essential.
I have found two possible approaches that might work, but I'm not sure what the best one is. The first is using filters to acheive the result, but I'm not sure if it will cope well with (a) the changing layouts and (b) keeping the audio without any artefacts when the layout changes.
The other approach I thought of would be exporting all frames to images, building new frames with imagemagick, and then layering the new frames on top of the audio like in this blog post.
Any suggestions on if either of these approaches is better, or any alternatives? Thanks!