20240046587. DEFORMING REAL-WORLD OBJECT USING AN EXTERNAL MESH simplified abstract (Snap Inc.)
Contents
DEFORMING REAL-WORLD OBJECT USING AN EXTERNAL MESH
Organization Name
Inventor(s)
Matan Zohar of Rishon LeZion (IL)
Brian Fulkerson of London (GB)
DEFORMING REAL-WORLD OBJECT USING AN EXTERNAL MESH - A simplified explanation of the abstract
This abstract first appeared for US patent application 20240046587 titled 'DEFORMING REAL-WORLD OBJECT USING AN EXTERNAL MESH
Simplified Explanation
Methods and systems are disclosed for performing operations involving the manipulation of videos using augmented reality (AR) elements. The patent application describes a process that includes the following steps:
- Receiving a video that contains a depiction of a real-world object.
- Generating a three-dimensional (3D) body mesh associated with the real-world object, which tracks the movement of the object across frames of the video.
- Determining the UV positions of the real-world object depicted in the video to obtain pixel values associated with these positions.
- Generating an external mesh and an associated AR element that represents the real-world object based on the pixel values obtained from the UV positions.
- Deforming the external mesh based on changes to the 3D body mesh and a deformation parameter.
- Modifying the video by replacing the real-world object with the AR element based on the deformed external mesh.
Potential applications of this technology:
- Augmented reality in video editing: This technology can be used to seamlessly integrate AR elements into videos, allowing for enhanced visual effects and storytelling.
- Virtual try-on: By tracking the movement of real-world objects, such as clothing or accessories, this technology can enable users to virtually try on different items in a video, enhancing the online shopping experience.
- Interactive advertising: Brands can use this technology to create interactive and engaging advertisements by replacing real-world objects in videos with AR elements that provide additional information or interactive features.
Problems solved by this technology:
- Seamless integration of AR elements: This technology solves the problem of accurately tracking the movement of real-world objects in videos and seamlessly replacing them with AR elements, creating a more immersive and realistic experience.
- Efficient video editing: By automating the process of generating and deforming meshes based on the movement of real-world objects, this technology streamlines the video editing process, saving time and effort.
Benefits of this technology:
- Enhanced visual effects: By incorporating AR elements into videos, this technology allows for the creation of visually stunning and immersive content.
- Improved user engagement: The ability to interact with AR elements in videos enhances user engagement and provides a more interactive and personalized experience.
- Streamlined video editing process: The automation of mesh generation and deformation simplifies the video editing process, making it more efficient and accessible to a wider range of users.
Original Abstract Submitted
methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3d) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; determining uv positions of the real-world object depicted in the video to obtain pixel values associated with the uv positions; generating an external mesh and associated augmented reality (ar) element representing the real-world object based on the pixel values associated with the uv positions; deforming the external mesh based on changes to the 3d body mesh and a deformation parameter; and modifying the video to replace the real-world object with the ar element based on the deformed external mesh.
