Overview of New Experimental Coding Tools by Sarah Parker
"Explore the latest coding tools in the realm of AV1 codec technology. From block partitions to transform schemes, this overview delves into the advancements shaping the next generation of coding techniques. Discover the flexible block partition schemes, extended transform partitioning, and data-driven transforms that are revolutionizing the industry."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
An Overview of New Experimental Coding Tools Sarah Parker Google
Overview Background Technical Overview Block Partitions Transform Coding Tools Intra Coding Tools Inter Coding Tools In-Loop Restoration Tools Conclusion
Overview Background Technical Overview Block Partitions Transform Coding Tools Intra Coding Tools Inter Coding Tools In-Loop Restoration Tools Conclusion
Background AV1 released in 2018 31% BDRATE reduction over VP9 Accounts for over 80% of the desktop market YouTube default on compatible browsers up to 720p AOMedia continues to explore new coding tools for the next generation codec
Overview Background Technical Overview Block Partitions Transform Coding Tools Intra Coding Tools Inter Coding Tools In-Loop Restoration Tools Conclusion
Highly Flexible Block Partitions Current Scheme: 128x128 R R R R
Highly Flexible Block Partitions Proposed Scheme: R R R R R 128x128 R R R R R
Overview Background Technical Overview Block Partitions Transform Coding Tools Intra Coding Tools Inter Coding Tools In-Loop Restoration Tools Conclusion
New Transform Partitioning Current scheme: Rectangular Blocks Square Blocks
New Transform Partitioning New extended partitioning scheme: Same partitions available for both square and rectangular blocks 0.6% coding gain, currently inter frames only
Data Driven Transforms Add MDTX: Transforms learned from collected block data Different schemes for inter and intra blocks
Data Driven Transforms for Inter Inter RDOTs: 2D combinations of a MDTX, flipped MDTX, and DCT 6 additional, 22 total for Inter DCT ADST FLIPADST IDTX MDTX FLIPMDTX Hor \ Ver DCT_DCT ADST_DCT FLIPADST_DCT H_DCT DCT MDTX_INTER_2 MDTX_INTER_5 DCT_ADST ADST_ADST FLIPADST_ADST H_ADST ADST DCT_FLIPADST ADST_FLIPADST FLIPADST_FLIPADST H_FLIPADST FLIPADST V_DCT V_ADST V_FLIPADST IDTX IDTX MDTX MDTX_INTER_3 MDTX_INTER_1 MDTX_INTER_7 FLIPMDTX MDTX_INTER_6 MDTX_INTER_8 MDTX_INTER_4
Data Driven Mode Dependent Transforms for Intra Intra RDOTs: MDTX/MDTX, MDTX/DCT, DCT/MDTX for each mode DCT ADST IDTX MDTX (1 per mode) Hor \ Ver DCT_DCT ADST_DCT H_DCT DCT DCT_MDTX DCT_ADST ADST_ADST H_ADST ADST V_DCT V_ADST IDTX IDTX MDTX (1 per mode) MDTX_DCT MDTX_MDTX
Non-separable Mode-dependent RDOTs for Intra Motivation: intra residual blocks exhibit patterns that are not horizontal/vertical Non-separable transforms tend to better capture these patterns Goal: introduce a 2D non-sep. transform for each intra mode D157_PRED (first 16 tx. functions) D67_PRED (first 16 tx. functions)
Non-separable Mode-dependent RDOTs for Inra Motivation: intra residual blocks exhibit patterns that are not horizontal/vertical Non-separable transforms tend to better capture these patterns Goal: introduce a 2D non-sep. transform for each intra mode Constrained to 8x8, 4x8 and 8x4 for complexity reasons Exploring second order transforms for larger block sizes D157_PRED (first 16 tx. functions) D67_PRED (first 16 tx. functions)
RD Results Separable transforms only Sep, & non-sep. transforms Lowres BD rate / instructions Midres BD rate / instructions Lowres BD rate / instructions Midres BD rate / instructions -0.699% / 308.5% -0.485% / 273.6% -0.791% / 337.8% -0.875% / 296.5% Inter + intra Inter + intra -0.447% / 284.1% -0.243% / 255.4% -0.447% / 284.1% -0.243% / 255.4% Inter only Inter only -0.346% / 242.8% -0.652% / 238.4% -0.273% / 219.6% -0.243% / 218.2% Intra only Intra only -1.086% / 435.9% -1.125% / 463.4% -0.637% / 348.5% -0.443% / 377.1% Key frames Key frames
Overview Background Technical Overview Block Partitions Transform Coding Tools Intra Coding Tools Inter Coding Tools In-Loop Restoration Tools Conclusion
Adaptive Recursive Filtering Based Intra Prediction Fit the filter coefficients adaptively for each block being predicted: training region from already reconstructed pixels (available in both encoder and decoder) regularization coefficient filter coefficients used to predict the target block
Adaptive Recursive Filtering Based Intra Prediction Available Filter Configurations , , 2-tap filters: full context , , 3-tap filters: left context , 4-tap filters: top context 54 modes 18 filter configurations 3 training regions =
Adaptive Recursive Filtering Based Intra Prediction Current experiment results: Test Settings midres (keyframe- only) hdres (keyframe- only) -1.207% -1.094% 7 adaptive modes 5 adaptive modes -0.954% -1.125% 3 adaptive modes -0.721% -0.845%
ML Based Entropy Coding for Intra Prior Art: Estimate the symbol probability distribution using lookup tables. Lookup tables indexed by contexts Shared initialization Lookup table Lookup table update update Entropy Coding Entropy Coding symbol symbol bitstream
ML Based Entropy Coding for Intra Disadvantages: To keep the size of the tables small, we cannot afford too many contexts. The contexts have to be discrete / categorical. Proposed idea: Replace lookup tables with ML models such as logistic regression or neural network models Use the ML models as classifiers to estimate symbol probabilities
ML Based Entropy Coding for Intra Shared initialization ML Model ML Model update update Entropy Coding Entropy Coding symbol symbol bitstream
ML Based Entropy Coding for Intra Use logistic regression model to estimate the probabilities of intra prediction modes Use top and left neighbor blocks modes as input features 0.6% coding gains for keyframes
Overview Background Technical Overview Block Partitions Transform Coding Tools Intra Coding Tools Inter Coding Tools In-Loop Restoration Tools Conclusion
Inter Coding Tools Motion vector prediction Illumination compensation Flexible motion vector precision
Overview Background Technical Overview Block Partitions Transform Coding Tools Intra Coding Tools Inter Coding Tools In-Loop Restoration Tools Conclusion
In-Loop Restoration Tools Compressed Bitstream Predictor Info Best Predictor Inter / Intra Reconstruct Blocks Loss introduced T Q Coefficients Restore Frame Source block Prediction Residue Deblocking, Others [Many new opportunities] Restoration Info Reference Frame store Several tools in AV1 for in-loop restoration other than Deblocking CDEF (CLPF, Deringing) Loop Restoration (Wiener, Dual Self-guided filter)
In-Loop Restoration Tools Compressed Bitstream Predictor Info Best Predictor Inter / Intra Reconstruct Blocks Loss introduced T Q Coefficients Restore Frame Source block Prediction Residue Deblocking, Others [Many new opportunities] Restoration Info Reference Frame store Replace CDEF and loop filter with CNN 3-4% coding gain Still exploring complexity reduction Non-Separable Wiener Filters
Overview Background Technical Overview Block Partitions Transform Coding Tools Intra Coding Tools Inter Coding Tools In-Loop Restoration Tools Conclusion
Conclusion About 2% additive BDRATE reduction over AV1 Most experiments still a work in progress Many more experiments to come, suggestions welcome!
