In a review comment from an earlier version of my patchset Implement SEI parsing for QSV decoders
there was a concern that the payload data retrieved with MFXVideoDECODE_GetPayload() might
need to be re-ordered and matched to output frames by timestamps - especially for cases where
the encoding order of frames does not match the display order (like with B-frames).
What is known and guaranteed is that MFXVideoDECODE_DecodeFrameAsync() is returning frames
(surfaces) in display order, so the concern was that GetPayload might return payload data
in encode order.
To confirm or deny this concern, I started with a test file.. ...and verified that it has unordered B-frames.
The output is from ffprobe, showing frames (frames are in presentation order) and you can see that the packet position in the bitstream is unordered.
I also checked the other way round by letting ffprobe output packets (in encoding order):
Here you can see the PTS time being unordered.
Now, when I run ffmpeg with my GetPayload implementation, it provides the following output:
When you divide the timestamps by 90000, you get exactly the values that ffprobe shows:
But those first few frames are encoded in a different order.
Nonetheless gives us MSDK always the right payloads. So, it doesn’t seem that any re-ordering would be required. (I can check H.264 later, but I don’t think that it would be different)
For this, I added a warning output to the code like this:
ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload);
if (ret != MFX_ERR_NONE) {
av_log(avctx, AV_LOG_WARNING, "error getting SEI payload: %d \n", ret);
return 0;
}
if (ts != surface->Data.TimeStamp) {
av_log(avctx, AV_LOG_WARNING, "GetPayload timestamp (%llu) does not match surface timestamp: (%llu)\n", ts, surface->Data.TimeStamp);
}Then I ran tests over a collection of test files and there wasn't a single case where the warning was output.
I was able to confirm that MSDK is performing the re-ordering, here’s the implementation for HEVC:
This is actually a pretty weird logic where the “isUsed” field is not about whether used or not – it’s rather a counter by which the sei messages are sorted in display order.
That provides the following behavior: When you always – after a successful decode call – execute GetPayload() in a loop until no more are available, then you will automatically get the right ones for that decoded frame (without need for sync completion).
This confirms that my patchset has been correct and a re-ordering of payload data is not necessary.