Meta platforms technologies, llc (20250157130). Distortion-Free Passthrough Rendering for Mixed Reality

Distortion-Free Passthrough Rendering for Mixed Reality

Organization Name

meta platforms technologies, llc

Inventor(s)

Andrey Tovchigrechko of Saratoga CA US

Alexander Sorkine Hornung of Zurich CH

Sebastian Sztuk of Virum DK

Remo Ziegler of Uster CH

Oskar Linde of San Carlos CA US

Ricardo da Silveira Cabral of Zurich CH

Alex Locher of Oberrohrdorf CH

Anirud Thyagharajan of Zurich CH

Artem Filatov of Thalwil CH

Fabian Langguth of Wädenswil CH

Ngoc Minh Dang of Basel CH

Salah Eddine Nouri of Zurich CH

Paul Timothy Furgale of Thalwil CH

Distortion-Free Passthrough Rendering for Mixed Reality

This abstract first appeared for US patent application 20250157130 titled 'Distortion-Free Passthrough Rendering for Mixed Reality

Original Abstract Submitted

a device may access an input image of a real-world scene captured by a camera from a camera viewpoint. the device may render, from the camera viewpoint, an inpainting image of a background of the real-world scene based on a 3d reconstruction model of the background of the real-world scene, wherein the 3d reconstruction model is generated using previously-captured images and previously-generated depth estimates. the device may generate a depth estimate of the real-world scene and identify, based on the depth estimate, a first set of pixel locations and a second set of pixel locations in a passthrough image to be rendered. the device may render, from a viewpoint of an eye of a user, the passthrough image based on the input image, the inpainting image, and the depth estimate. first pixel values for the first set of pixel locations in the passthrough image are sampled from the input image, and second pixel values for the second set of pixel locations in the passthrough image are sampled from the inpainting image.