~ Movie Metric Benchmark Challenge ~
You think that your codec is the best : prove that ... !!?
A - Introduction
The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...
B - Rules
Rule 1 : Source
You must use this source for the test : Harry Potter II trailer, 720*576 MPEG2 MP@ML 4.5 Mbps
HPII Part1 winrar, HPII Part2 winrar, HPII Part3 winrar, HPII Part4 winrar & HPII Part5 winrar
Rule 2 : Pre-process and encoding
You must use DGindex with 32-bit SSE/MMX iDCT algorithm for .d2v project and DGDecode.dll for MPEG2 decoding in avisynth scipt. You must use VirtualDub or VirtualDubMod in "fast compress" mod for VFW codec. You must use YV12 space color for complete process with all codec. You must use this avisynth script for make encoding:
source=Mpeg2Source("C:/.../HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)
Return(source)
Rule 3 : Bitrate
You must use 2 bitrates for encoding. The first use XviD in quant8 for reference, it's the "Streaming Quality". The second use XviD in quant4 for reference, it's the "1CDR Quality". You can use container of your choice for all Video Elementary Stream.
Streaming: 450 Kbps for Video Elementary Stream and container or 6750 Ko with +/- 0.30 % for tolerance. The size must be in [6730 Ko - 6770 Ko] interval.
1CDR: 900 Kbps for Video Elementary Stream and container or 13500 Ko with +/- 0.30 % for tolerance. The size must be in [13460 Ko - 13540 Ko] interval.
Rule 4 : Metric Tests
SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM
Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))?/ (NFrame*width*height)
Overall PSNR = 10 * Log ( 255?/ MSE)
Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))?/ (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255?/ MSE) / Nframe
Here the test archives if you want all original Avisynth script for metric test.
SSIM vs PSNR
"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."
"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."
Here last AviSynth plugin for VSSIM in YV12
Average PSNR vs Overall PSNR
It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.
Here last AviSynth plugin for Overall PSNR in YV12
You must mux or remux x264, VP6, VP7, WMV9, DivX6 and XviD in avi container (with VDM for example) to solve problem with color space output, synchronisation or acquisition for metric test. You must use AviSource in avisynth for VP6, VP7 and WMV9. You must use DirectShowSource in avisynth for MPEG4 ASP, MPEG4 AVC and RV10. You must change trim for frame synchronization or for clip length if you have some probleme with SSIM plugin. You must use these AviSynth type script for metric test:
# --> Video Opening <--
source=Mpeg2Source("G:/.../HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)
video=DirectShowSource("G:/.../DivX6-450.avi",fps=25)
# --> PSNR analysis <--
return compareYV12(video,source,"YUV","PSNR-DivX6-450.log")
# --> Video Opening <--
source=Mpeg2Source("G:/.../HPII.d2v",idct=2)
source=Trim(source,70,3144)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)
video=DirectShowSource("G:/.../RV10-450.mkv",fps=25)
# --> PSNR analysis <--
return SSIM(source,video,"results.csv","SSIM-RV10-450.txt",lumimask=true)
C - Results
At this time here my result ... but done better if you can ...
source=Mpeg2Source("C:/.../HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)
Return(source)
Rule 3 : Bitrate
You must use 2 bitrates for encoding. The first use XviD in quant8 for reference, it's the "Streaming Quality". The second use XviD in quant4 for reference, it's the "1CDR Quality". You can use container of your choice for all Video Elementary Stream.
Streaming: 450 Kbps for Video Elementary Stream and container or 6750 Ko with +/- 0.30 % for tolerance. The size must be in [6730 Ko - 6770 Ko] interval.
1CDR: 900 Kbps for Video Elementary Stream and container or 13500 Ko with +/- 0.30 % for tolerance. The size must be in [13460 Ko - 13540 Ko] interval.
Rule 4 : Metric Tests
SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM
Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))?/ (NFrame*width*height)
Overall PSNR = 10 * Log ( 255?/ MSE)
Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))?/ (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255?/ MSE) / Nframe
Here the test archives if you want all original Avisynth script for metric test.
SSIM vs PSNR
"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."
"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."
Here last AviSynth plugin for VSSIM in YV12
Average PSNR vs Overall PSNR
It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.
Here last AviSynth plugin for Overall PSNR in YV12
You must mux or remux x264, VP6, VP7, WMV9, DivX6 and XviD in avi container (with VDM for example) to solve problem with color space output, synchronisation or acquisition for metric test. You must use AviSource in avisynth for VP6, VP7 and WMV9. You must use DirectShowSource in avisynth for MPEG4 ASP, MPEG4 AVC and RV10. You must change trim for frame synchronization or for clip length if you have some probleme with SSIM plugin. You must use these AviSynth type script for metric test:
# --> Video Opening <--
source=Mpeg2Source("G:/.../HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)
video=DirectShowSource("G:/.../DivX6-450.avi",fps=25)
# --> PSNR analysis <--
return compareYV12(video,source,"YUV","PSNR-DivX6-450.log")
# --> Video Opening <--
source=Mpeg2Source("G:/.../HPII.d2v",idct=2)
source=Trim(source,70,3144)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)
video=DirectShowSource("G:/.../RV10-450.mkv",fps=25)
# --> PSNR analysis <--
return SSIM(source,video,"results.csv","SSIM-RV10-450.txt",lumimask=true)
C - Results
At this time here my result ... but done better if you can ...
|---------------|-----------|------------|----------|---------|---------| | Codec | P-Process | ES Bitrate | Size | OPSNR | SSIM | |---------------|-----------|------------|----------|---------|---------| | DivX | PP4 | 446 kbps | 6755 Ko | 39.7552 | 64.44 | | XviD | PP4 | 446 kbps | 6760 Ko | 39.6601 | 64.36 | | Nero AVC | PP0 | 446 kbps | 6750 Ko | 40.6388 | 69.53 | | x264 | PP0 | 446 kbps | 6760 Ko | 40.4674 | 68.45 | | | PP2 | 446 kbps | 6749 Ko | 40.7279 | 68.84 | | VP6 | PP4 | 447 kbps | 6769 Ko | 40.2892 | 66.91 | | WMV9 | PP1 | 446 kbps | 6746 Ko | 39.5186 | 64.16 | | RV10 | HF1 | 446 kbps | 6743 Ko | 40.0457 | 65.52 | |---------------|-----------|------------|----------|---------|---------| | DivX | PP4 | 896 kbps | 13508 Ko | 42.7455 | 77.93 | | XviD | PP4 | 896 kbps | 13498 Ko | 42.6654 | 77.95 | | Nero AVC | PP0 | 896 kbps | 13509 Ko | 43.2847 | 80.13 | | x264 | PP0 | 896 kbps | 13498 Ko | 43.1386 | 79.32 | | | PP2 | 895 kbps | 13477 Ko | 43.2436 | 79.33 | | VP6 | PP4 | 895 kbps | 13481 Ko | 42.9933 | 78.41 | | WMV9 | PP1 | 896 kbps | 13493 Ko | 42.5286 | 76.74 | | RV10 | HF1 | 896 kbps | 13493 Ko | 42.7431 | 77.73 | |---------------|-----------|------------|----------|---------|---------|
MPEG4 ASP : DivX6
Coder: DivX Codec Fusion Build 1461
Setting: 2 pass, Unconstrained, Insame Quality, Multi-Bframe, H263 optmized, BM = 0.1
decoder: DivX decoder with PP4
files: DivX 450 Kbps & DivX 900 Kbps
MPEG4 ASP : XviD
Coder: Koepi XviD 1.1.0 beta1
Setting: 2 pass, Unconstrained, Motion Ultra-High, VHQ4, BVHQ, Chroma Motion, Trellis, Adaptative Quantisation, GMC, BFrame 2/1.50/1.00
decoder: XviD decoder with PP4
files: XviD 450 Kbps & XviD 900 Kbps
MPEG4 AVC : Nero Digital AVC
Coder: Ateme encavc beta4a
Setting: 2 pass, Max Definition AVC, Quality Extra, Max Vect Range [-512;+512], 16 Ref frames, 3 Bframes, CABAC, Bidirectionnal Pred, MPart, WPred, Inloop deblock strength=0 not adaptative, Chroma, psy low
decoder: Nero Digital AVC decoder
files: Nero AVC 450 Kbps & Nero AVC 900 Kbps
MPEG4 AVC : x264
Coder: x264 VFW rev 184
Setting: 3 pass, variabilty 75%, Quality Max, 15 Ref frames, 3 Bframes temporal, CABAC, WPred, Inloop deblock strength=0
decoder: Ateme dec
files: x264 450 Kbps & x264 900 Kbps
One2Tech : VP7
Coder: VP7 VFW Codec 7.0.0.4
Setting: 2 pass, best quality, VBR, Noisered=0, Sharpeness=0, MinQ 8/16, MaxQ 56, Vbr bias 70, Vbr min 10, Vbr max 200, Undershoot 95%
decoder: VP7 decoder PP2
files: VP7 450 Kbps & VP7 900 Kbps
One2Tech : VP6
Coder: VP6 VFW Codec 6.4.1.0
Setting: 2 pass, best quality, VBR, Noisered=0, Sharpeness=0, MinQ 6/12, MaxQ 56, Vbr bias 70, Vbr min 30, Vbr max 200, Undershoot 95%
decoder: VP6 decoder PP4
files: VP6 450 Kbps & VP6 900 Kbps
Microsoft : WMV9
Coder: WMV9 VCM
Setting: 2 pass VBR, best quality, decoder complex
decoder: WMV3 DMO decoder PP1
files: WMV9 450 Kbps & WMV9 900 Kbps
RealNetwork : RV10
Coder: erv4 10.0.0.2
Setting: 2 pass, EHQ 100, old Rate Control, chroma
decoder: RV10 decoder HFE1
files: RV10 450 Kbps & RV10 900 Kbps
MPEG2 MP@ML : TMPGEnc
find size for metric equivalence with best codec for quality "1CDR"
"Quality" must be at ~ 43 dB for OPSNR or at ~ 80 for SSIM
files: MPEG2 1560 Kbps
D - Graph
Here the test archives if you want original log files or make yourself Frames PSNR Graph with JXvidStatsViewerW
HPII trailer 450 Kbps
Nero AVC vs VP7
Nero AVC vs x264
XviD vs DivX6
HPII trailer 900 Kbps
Nero AVC vs VP7
Nero AVC vs x264
XviD vs DivX6
E - Conclusion
At this time for this trailer
H264 vs VP7 vs x264
VP7 is just little better than Nero H264 Main Profil and better than x264 for OPSNR test. Nero H264 with psy mode is better than VP7 and VP7 better than x264 for SSIM. x264 is the best Open Source Codec and XviD can't fight with the last version.
VP6 vs RV10
VP6 is always better than RV10 for OPSNR or SSIM.
XviD vs DivX vs WMV9
For the first time DivX is better than XviD for OPSNR and SSIM. MPEG4 ASP class codec (XviD or DivX) are always better than WMV9.
MPEG2 vs all
The best codecs (H264 or VP7) are 42% better than MPEG2 MP@ML: MPEG2 MP@ML with good encoder is always a good codec.
Annexe - Update
03.04.05
- x264 rev 184
- VP7 7.0.0.4
__________________