18121268. PLOTTING BEHIND THE SCENES WITH LEARNABLE GAME ENGINES simplified abstract (Snap Inc.)

PLOTTING BEHIND THE SCENES WITH LEARNABLE GAME ENGINES

Organization Name

Snap Inc.

Inventor(s)

Willi Menapace of Santa Monica CA (US)

Aliaksandr Siarohin of Los Angeles CA (US)

Sergey Tulyakov of Santa Monica CA (US)

PLOTTING BEHIND THE SCENES WITH LEARNABLE GAME ENGINES - A simplified explanation of the abstract

This abstract first appeared for US patent application 18121268 titled 'PLOTTING BEHIND THE SCENES WITH LEARNABLE GAME ENGINES

Simplified Explanation: This framework trains neural models from annotated videos to create a Learnable Game Engine (LGE) that can render environments from controllable viewpoints and enable gameplay through high- and low-level action sequences.

Key Features and Innovation:

• Trains neural models from annotated videos to generate a Learnable Game Engine (LGE).
• LGE maintains states of the scene, objects, and agents, and models the logic of the game and rules of physics.
• Enables gameplay through high- and low-level action sequences.
• Unlocks a director's mode for playing the game by specifying high-level actions and goals for agents using text-based instructions.
• Uses diffusion-based animation model for director's mode and compositional neural radiance field representation for rendering the environment.

Potential Applications: This technology can be applied in the development of interactive video games, virtual reality simulations, training simulations, and educational tools.

Problems Solved: This technology addresses the need for more interactive and controllable game engines that can be trained from annotated videos and enable gameplay through high- and low-level actions.

Benefits:

• Enhanced user experience in gameplay through controllable viewpoints and high- and low-level action sequences.
• Simplified game development process by training neural models from annotated videos.
• Versatile application in various industries such as gaming, virtual reality, training simulations, and education.

Commercial Applications: Potential commercial applications include the development of interactive video games, virtual reality experiences, training simulations for various industries, and educational tools for immersive learning experiences.

Prior Art: Readers interested in prior art related to this technology can explore research on neural network-based game engines, video annotation techniques for training models, and interactive gameplay systems.

Frequently Updated Research: Stay updated on the latest advancements in neural network training for game engines, video annotation techniques for model training, and interactive gameplay systems.

Questions about the Technology: 1. How does the framework train neural models from annotated videos to create a Learnable Game Engine? 2. What are the potential applications of this technology in various industries?

Original Abstract Submitted

A framework trains game-engine-like neural models from annotated videos to generate a Learnable Game Engine (LGE) that maintains states of the scene, objects and agents in it, and enables rendering the environment from a controllable viewpoint. The LGE models the logic of the game and the rules of physics, making it possible for the user to play the game by specifying both high- and low-level action sequences. The LGE also unlocks a director's mode where the game is played by plotting behind the scenes, specifying high-level actions and goals for the agents using text-based instructions. To implement the director's mode, a trained diffusion-based animation model navigates the scene using high-level constraints, to enable play against an adversary, and to devise the strategy to win a point. To render the resulting state of the environment and its agents, a compositional neural radiance field (NeRF) representation is used in a synthesis model.