So I was reading Hacker News and decided to read the comments in the thread about H.265 being approved. Pretty close to the top was this comment about VP9, Google's future video format. I have some words of my own about it and other future formats at the bottom of this post, but what jumped out from the comment to me was this part:
Many have already implemented VP8 (which is also slightly better than h.264 at this point)
The comparison linked to back up that statement is faulty for several reasons, such as not providing the source material used (hell, he doesn't even name the source material), exact encoding settings used (no, some random profiles are not enough), not providing the resulting encodes, only providing a single image for the whole comparison, not providing encoder logs... I think you get the idea. So I decided to do a quick comparison of my own, and to do it right. Here are the results.
Testing with just a single clip has its downsides - most test clips can benefit from encoders that have features X and/or Y. The clip currently used in this test (park Joy) seems to favor encoders with large block size, and VP8 as a format fares worse in this regard compared to H.264. My original reason for choosing Park Joy was because x264 developer Dark_Shikari said that "it shouldn't bias too heavily towards any one encoder like many of the other standard test clips will", but apparently this isn't so correct (see comments at bottom). If you're interested in technical properties of VP8 compared to H.264, you might be interested in this analysis. It is quite likely that multiple test encodes will not affect the conclusion much, though, seeing as VP8 as a format is weaker than H.264. The situation could be different if H.264 didn't have such a top-notch encoder as x264 or if we were only comparing to H.264 Baseline, but this comparison is about maximum[1] quality obtainable with each format (using the best encoders available).
- Test Clip: park_joy_1080p50.y4m - Download here
- Encode Target: "Best quality" 2-pass 13600 kbps encode at 1080p50.
- Test Platform: i7-2600k @ ~4.4 GHz, Windows 7 64-bit (other details don't really matter)
- H.264 Encoder: x264 r2245, 64-bit - Download here
- VP8 Encoder: vpxenc, 64-bit (libvpx 1.1.0) - Download here
vpxenc -w 1920 -h 1080 --fps=50/1 --best -p 2 --fpf=vp8.stats --target-bitrate=13600 --end-usage=vbr --auto-alt-ref=1 --minsection-pct=5 --maxsection-pct=800 --lag-in-frames=16 --kf-min-dist=0 --kf-max-dist=250 --static-thresh=0 --drop-frame=0 --min-q=0 --max-q=60 -t 7 -o park_joy_vp8.webm park_joy_1080p50.y4m
x264 --input-res 1920x1080 --fps 50 --preset veryslow --tune film --pass 1 --stats h264.stats --bitrate 13600 -i 1 -I 250 -o NUL park_joy_1080p50.y4m
x264 --input-res 1920x1080 --fps 50 --preset veryslow --tune film --pass 2 --stats h264.stats --bitrate 13600 -i 1 -I 250 -o park_joy_h264.mkv park_joy_1080p50.y4m
- A little background on me: I have about five years of experience in processing and encoding digital video.
- The vpxenc settings are based on the "2-Pass Best Quality VBR Encoding" settings found on the WebM homepage here. x264 settings are based on my own experience (though they're not really anything special in this case).
- I decided to use
--preset veryslowinstead of--preset placebofor x264 because placebo is really damn slow and so that both encoders would do a fast first pass (--preset placebodisables this). - As you might deduct from the command line parameters above, x264's preset and tune system makes things quite convenient. When I do actual encoding work, I usually start with
--preset veryslow --tune [something]and only tweak a few settings beyond that. - VP8 encoding used about ~80% of the CPU at the recommended threads setting (cores - 1) on the second pass.
- H.264 encoding used about ~95% of the CPU (with threads automatically set to 12 by x264) on the second pass.
- VP8 [Download] - 13144 kbps (16436KB), first pass: 25.94 fps, second pass: 2.40 fps
- H.264 [Download] - 13498 kbps (16483KB), first pass: 39.12 fps, second pass: 4.93 fps (full command line output below)
- Screenshot Comparison - As the clip is quite similar throughout, I only took one pair of shots - if you want to see more, download and the watch the encoded videos yourself.
H.264 encoded with the latest x264 offers notably higher quality while encoding almost twice as fast as VP8 encoded with the latest libvpx offering. If you see a test claiming that VP8 is better than H.264 quality-wise, it is very likely that the comparison was done poorly, either by mistake or intentionally. I very much recommend reading this article by x264 developer Jason Garrett-Glaser on the subject.
I am very much looking forward to what future brings us in the field of video formats. The keyword here is future - even though H.265 is "approved" now, it'll be a while before we get actually usable encoding and decoding implementations (reference encoders are hardly great examples of what the format will be truly capable of - implementation matters a lot[2]). Other formats that have been drummed up recently, namely On2's (Google's) VP9 and Xiph's Daala, are certainly interesting as well, but remember to take any wild claims with a large grain of salt: On2 is pretty infamous when it comes to overmarketing their products (in case of VP9, a while back they said VP9 is "only ~7% behind H.265 (when compared to HEVC JM)" - the JM here means reference encoder) and Daala doesn't have much beyond its big words going for it at the moment.
Seeing as both also aim to be royalty-free (so essentially "patent-free"), beating H.265 will be no easy task, considering how much of a patent minefield the field of video encoding is. While I'd love a patent- and royalty-free video format to offer the highest quality compression you can find, I wouldn't hold my breath for one.
Any Not-H.265 format will also find it much more hard to get hardware support considering that H.265 is an "industry standard" - so even if they offered better quality than H.265, they could end up struggling when it comes to widespread adoption. If they can't offer better quality, then even more so. The most likely scenario will likely end up being very similar to the situation with H.264, VP8 and Theora today.
[1] Technically we could go even higher with H.264 than in this test by using something like the High 10 Profile (and placebo preset in x264), but most widespread "high quality" H.264 usage is limited to High Profile, so that's what we're using here.
[2] For those wondering why I didn't link to a test including the reference encoder, I couldn't really find any. Most likely because the reference encoder is very, very slow - x264 is approx. 50 times faster than it at the same quality level!
It does not.
So look for parts where they do better or worse— and point them out. It would at least let you discover things like this clip or that clip is really good with one encoder or the other.
But if you're going to purport to report a winner then there really isn't a replacement for a lot of work. :(
I apologize if I've insulted you— it wasn't my intention. I'm expressing my earnest opinion. One single clip comparison says X is better, one single clip comparison says Y is better. The elephant in the lab is "maybe the clip determined the outcome?" and nothing has been done to answer that question in either case.