20240046590. Reconstruction of Essential Visual Cues in Mixed Reality Applications simplified abstract (Meta Platforms Technologies, LLC)

Reconstruction of Essential Visual Cues in Mixed Reality Applications

Organization Name

Meta Platforms Technologies, LLC

Inventor(s)

James Allan Booth of Pacifica CA (US)

Gaurav Chaurasia of Meilen (CH)

Alexandru-Eugen Ichim of Rüschlikon (CH)

Alex Locher of Oberrohrdorf (CH)

Gioacchino Noris of Zurich (CH)

Alexander Sorkine Hornung of Zurich (CH)

Manuel Werlberger of Zurich (CH)

Reconstruction of Essential Visual Cues in Mixed Reality Applications - A simplified explanation of the abstract

This abstract first appeared for US patent application 20240046590 titled 'Reconstruction of Essential Visual Cues in Mixed Reality Applications

Simplified Explanation

The abstract describes a mixed reality simulation system that uses a console and a head-mounted device (HMD) to capture stereoscopic images from the real world. The system preprocesses the images to enhance contrast and extracts features such as edges or corners. It then generates 2D polylines for each feature and triangulates between the polylines in the right and left side images to create 3D polylines. The system interpolates or extrapolates vertices to generate a geometric reconstruction of the real-world environment. Textures from the real-world environment can be mapped onto the geometric representation.

• The system captures stereoscopic images from the real world using outward-facing stereoscopic cameras mounted on the HMD.
• It preprocesses the images to maximize contrast and extracts features such as edges or corners.
• 2D polylines are generated for each feature.
• Triangulation is performed between the 2D polylines in the right and left side images to generate 3D polylines.
• The system interpolates or extrapolates vertices to create a geometric reconstruction of the real-world environment.
• Textures derived from the real-world environment can be mapped onto the geometric representation.

Potential applications of this technology:

• Mixed reality simulations and experiences.
• Virtual tours of real-world environments.
• Training and education in various fields such as architecture, engineering, and medicine.
• Gaming and entertainment experiences.

Problems solved by this technology:

• Accurate and efficient reconstruction of real-world environments in mixed reality simulations.
• Enhancing the visual quality and realism of mixed reality experiences.
• Enabling the mapping of textures from the real world onto virtual representations.

Benefits of this technology:

• Immersive and realistic mixed reality experiences.
• Improved training and education through interactive simulations.
• Enhanced visual quality and realism in virtual environments.
• Efficient and accurate reconstruction of real-world environments.

Original Abstract Submitted

a mixed reality (mr) simulation system includes a console and a head mounted device (hmd). the mr system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the hmd. the mr system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. for each feature, the mr system generates one or more two-dimensional (2d) polylines. then, the mr system triangulates between 2d polylines found in right side images and corresponding 2d polylines found in left side images to generate a set of 3d polylines. the mr system interpolates between 3d vertices included in the 3d polylines or extrapolates additional 3d vertices, thereby generating a geometric reconstruction of the real-world environment. the mr system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.