17951919. HYBRID NEURAL NETWORK BASED END-TO-END IMAGE AND VIDEO CODING METHOD simplified abstract (Apple Inc.)

HYBRID NEURAL NETWORK BASED END-TO-END IMAGE AND VIDEO CODING METHOD

Organization Name

Apple Inc.

Inventor(s)

Alican Nalci of Cupertino CA (US)

Alexandros Tourapis of Los Gatos CA (US)

Hsi-Jung Wu of San Jose CA (US)

Jiefu Zhai of Sunnyvale CA (US)

Jingteng Xue of Cupertino CA (US)

Jun Xin of San Jose CA (US)

Mei Guo of San Jose CA (US)

Xingyu Zhang of Cupertino CA (US)

Yeqing Wu of Cupertino CA (US)

Yunfei Zheng of Santa Clara CA (US)

Jean Begaint of Cupertino CA (US)

HYBRID NEURAL NETWORK BASED END-TO-END IMAGE AND VIDEO CODING METHOD - A simplified explanation of the abstract

This abstract first appeared for US patent application 17951919 titled 'HYBRID NEURAL NETWORK BASED END-TO-END IMAGE AND VIDEO CODING METHOD

Simplified Explanation

Improved neural-network-based image and video coding techniques are presented, which incorporate both traditional coding tools and neural-network-based tools. The host coding tools may include established video coding standards like H.266 (VVC). The techniques involve partitioning the source frames and selecting either host or neural-network-based tools for each partition. Coding parameter decisions are made based on the partitioning and tool selection. Rate control for both host and neural network tools is combined. The output of the neural network undergoes multi-stage processing using a checkerboard prediction pattern.

• The patent application proposes improved techniques for image and video coding using neural networks.
• These techniques combine traditional coding tools with neural-network-based tools.
• The host coding tools may include established video coding standards like H.266 (VVC).
• Source frames are partitioned, and the appropriate coding tool (host or neural network) is selected for each partition.
• Coding parameter decisions are made based on the partitioning and tool selection.
• Rate control for both host and neural network tools is combined.
• The output of the neural network undergoes multi-stage processing using a checkerboard prediction pattern.

Potential Applications

The technology described in this patent application has potential applications in various fields, including:

• Video streaming platforms and services
• Video compression and storage systems
• Image and video editing software
• Virtual reality and augmented reality applications
• Video surveillance and security systems

Problems Solved

The patent application addresses the following problems:

• Traditional image and video coding techniques may not fully leverage the capabilities of neural networks.
• Neural network-based coding techniques may not integrate well with existing video coding standards.
• Efficiently combining host coding tools and neural network-based tools can be challenging.
• Rate control for both types of coding tools may require separate approaches.
• Enhancing the output of neural networks for image and video coding can be complex.

Benefits

The technology described in this patent application offers the following benefits:

• Improved image and video coding efficiency and quality.
• Integration of neural network-based tools with existing video coding standards.
• Enhanced flexibility in selecting coding tools based on partitioning.
• Combined rate control for host and neural network tools.
• Multi-stage processing of neural network output for improved prediction accuracy.

Original Abstract Submitted

Improved neural-network-based image and video coding techniques are presented, including hybrid techniques that include both tools of a host codec and neural-network-based tools. In these improved techniques, the host coding tools may include conventional video coding standards such H.266 (VVC). In an aspects, source frames may be partitioned and either host or neural-network-based tools may be selected per partition. Coding parameter decisions for a partition may be constrained based on the partitioning and coding tool selection. Rate control for host and neural network tools may be combined. Multi-stage processing of neural network output may use a checkerboard prediction pattern.