20240046587. DEFORMING REAL-WORLD OBJECT USING AN EXTERNAL MESH simplified abstract (Snap Inc.)

DEFORMING REAL-WORLD OBJECT USING AN EXTERNAL MESH

Organization Name

Snap Inc.

Inventor(s)

Matan Zohar of Rishon LeZion (IL)

Yanli Zhao of London (GB)

Brian Fulkerson of London (GB)

Riza Alp Guler of London (GB)

DEFORMING REAL-WORLD OBJECT USING AN EXTERNAL MESH - A simplified explanation of the abstract

This abstract first appeared for US patent application 20240046587 titled 'DEFORMING REAL-WORLD OBJECT USING AN EXTERNAL MESH

Simplified Explanation

Methods and systems are disclosed for performing operations involving the manipulation of videos using augmented reality (AR) elements. The patent application describes a process that includes the following steps:

• Receiving a video that contains a depiction of a real-world object.
• Generating a three-dimensional (3D) body mesh associated with the real-world object, which tracks the movement of the object across frames of the video.
• Determining the UV positions of the real-world object depicted in the video to obtain pixel values associated with these positions.
• Generating an external mesh and an associated AR element that represents the real-world object based on the pixel values obtained from the UV positions.
• Deforming the external mesh based on changes to the 3D body mesh and a deformation parameter.
• Modifying the video by replacing the real-world object with the AR element based on the deformed external mesh.

Potential applications of this technology:

• Augmented reality in video editing: This technology can be used to seamlessly integrate AR elements into videos, allowing for enhanced visual effects and storytelling.
• Virtual try-on: By tracking the movement of real-world objects, such as clothing or accessories, this technology can enable users to virtually try on different items in a video, enhancing the online shopping experience.
• Interactive advertising: Brands can use this technology to create interactive and engaging advertisements by replacing real-world objects in videos with AR elements that provide additional information or interactive features.

Problems solved by this technology:

• Seamless integration of AR elements: This technology solves the problem of accurately tracking the movement of real-world objects in videos and seamlessly replacing them with AR elements, creating a more immersive and realistic experience.
• Efficient video editing: By automating the process of generating and deforming meshes based on the movement of real-world objects, this technology streamlines the video editing process, saving time and effort.

Benefits of this technology:

• Enhanced visual effects: By incorporating AR elements into videos, this technology allows for the creation of visually stunning and immersive content.
• Improved user engagement: The ability to interact with AR elements in videos enhances user engagement and provides a more interactive and personalized experience.
• Streamlined video editing process: The automation of mesh generation and deformation simplifies the video editing process, making it more efficient and accessible to a wider range of users.

Original Abstract Submitted

methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3d) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; determining uv positions of the real-world object depicted in the video to obtain pixel values associated with the uv positions; generating an external mesh and associated augmented reality (ar) element representing the real-world object based on the pixel values associated with the uv positions; deforming the external mesh based on changes to the 3d body mesh and a deformation parameter; and modifying the video to replace the real-world object with the ar element based on the deformed external mesh.