Meta platforms technologies, llc (20240119568). View Synthesis Pipeline for Rendering Passthrough Images simplified abstract

View Synthesis Pipeline for Rendering Passthrough Images

Organization Name

meta platforms technologies, llc

Inventor(s)

Andrey Tovchigrechko of SARATOGA CA (US)

Fabian Langguth of Wädenswil (CH)

Alexander Sorkine Hornung of Zurich (CH)

Oskar Linde of San Carlos CA (US)

Christian Forster of Zofingen (CH)

View Synthesis Pipeline for Rendering Passthrough Images - A simplified explanation of the abstract

This abstract first appeared for US patent application 20240119568 titled 'View Synthesis Pipeline for Rendering Passthrough Images

Simplified Explanation

The patent application describes a processor that generates segmented depth maps and meshes for different object types in a scene captured by an artificial reality device. The processor then generates eye-specific meshes for each user by compositing the segmented meshes based on segmentation information from a second image captured for each eye.

• Processor generates segmentation masks for object types based on first image
• Processor segments depth map into segmented depth maps for object types
• Processor generates meshes using segmented depth maps
• Processor captures second image for each eye of the user
• Processor generates segmentation information based on second image
• Processor warps meshes to generate eye-specific meshes
• Processor composites warped meshes based on segmentation information
• Processor renders output image for each eye using second image and eye-specific mesh

Potential Applications

This technology can be applied in virtual reality gaming, augmented reality applications, medical imaging, and industrial design.

Problems Solved

This technology solves the problem of accurately rendering virtual scenes for users by generating eye-specific meshes based on segmentation information.

Benefits

The benefits of this technology include improved visual quality, enhanced user experience, and more realistic virtual environments.

Potential Commercial Applications

This technology can be used in the development of virtual reality headsets, augmented reality glasses, medical imaging devices, and industrial design software.

Possible Prior Art

One possible prior art for this technology could be the use of depth maps and segmentation masks in computer graphics rendering techniques.

How does this technology handle occlusions between different object types in the scene?

The technology does not specifically address how occlusions between different object types are handled in the scene.

Can this technology be applied to real-time rendering in virtual reality applications?

The technology does not mention real-time rendering capabilities, so it is unclear if it can be applied to real-time rendering in virtual reality applications.

Original Abstract Submitted

a processor accesses a depth map and a first image of a scene generated using one or more sensors of an artificial reality device. the processor generates, based on the first image, segmentation masks respectively associated with a plurality of object types. the segmentation masks segment the depth map into a plurality of segmented depth maps respectively associated with the object types. the processor generates meshes using, respectively, the segmented depth maps. for each eye of the user, the processor captures a second image and generates, based on the second image, segmentation information. the processor warps the plurality of meshes to generate warped meshes for the eye, and then generates an eye-specific mesh for the eye by compositing the warped meshes according to the segmentation information. the processor renders an output image for the eye using the second image and the eye-specific mesh.