18251743. VOLUMETRIC PERFORMANCE CAPTURE WITH NEURAL RENDERING simplified abstract (GOOGLE LLC)

VOLUMETRIC PERFORMANCE CAPTURE WITH NEURAL RENDERING

Organization Name

GOOGLE LLC

Inventor(s)

Sean Ryan Francesco Fanello of San Francisco CA (US)

Abhi Meka of Redwood City CA (US)

Rohit Kumar Pandey of Mountain View CA (US)

Christian Haene of Berkeley CA (US)

Sergio Orts Escolano of San Francisco CA (US)

Christoph Rhemann of Marina Del Rey CA (US)

Paul Debevec of Culver City CA (US)

Sofien Bouaziz of Los Gatos CA (US)

Thabo Beeler of Zurich (CH)

Ryan Overbeck of San Francisco CA (US)

Peter Barnum of Mountain View CA (US)

Daniel Erickson of San Francisco CA (US)

Philip Davidson of Arlington MA (US)

Yinda Zhang of Palo Alto CA (US)

Jonathan Taylor of New York NY (US)

Chloe Legendre of Culver City CA (US)

Shahram Izadi of San Francisco CA (US)

VOLUMETRIC PERFORMANCE CAPTURE WITH NEURAL RENDERING - A simplified explanation of the abstract

This abstract first appeared for US patent application 18251743 titled 'VOLUMETRIC PERFORMANCE CAPTURE WITH NEURAL RENDERING

Simplified Explanation

The patent application describes a technique for capturing the performance of a subject in a three-dimensional (volumetric) manner using neural rendering. Here are the key points:

• The technique involves capturing images of the subject from multiple viewpoints and lighting conditions using a light stage, along with depth data obtained from infrared cameras.
• A neural network is used to extract features of the subject from the images based on the depth data.
• These features are then mapped into a texture space (UV texture space) for further processing.
• A neural renderer is employed to generate an output image of the subject from a desired viewpoint, ensuring that the illumination in the output image aligns with the target view.
• The neural renderer achieves this by resampling the features of the subject from the texture space to an image space, resulting in the generation of the output image.

Potential applications of this technology:

• Film and entertainment industry: This technique can be used to capture performances of actors or characters in a more realistic and immersive manner, enabling the creation of high-quality visual effects.
• Virtual reality (VR) and augmented reality (AR): By capturing volumetric performances, this technique can enhance the realism and interactivity of VR and AR experiences.
• Gaming: Game developers can utilize this technique to capture and render realistic character animations, improving the overall gaming experience.

Problems solved by this technology:

• Traditional methods of capturing performances often rely on markers or sensors attached to the subject, which can be cumbersome and limit the freedom of movement. This technique eliminates the need for physical markers or sensors.
• The use of neural rendering allows for more accurate and realistic rendering of the subject, including proper alignment of illumination with the target view.

Benefits of this technology:

• Improved realism: The use of neural rendering and volumetric capture techniques enhances the realism of captured performances, resulting in more immersive and engaging visual content.
• Increased efficiency: By eliminating the need for physical markers or sensors, the capture process becomes more efficient and less intrusive for the subject.
• Flexibility and freedom of movement: The technique allows for capturing performances with natural movement and without restrictions, enabling more authentic and dynamic representations.

Original Abstract Submitted

Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.