Roblox Corporation (20240428492). ROBUST FACIAL ANIMATION FROM VIDEO AND AUDIO

ROBUST FACIAL ANIMATION FROM VIDEO AND AUDIO

Organization Name

Roblox Corporation

Inventor(s)

Iñaki Navarro of Zurich (CH)

Dario Kneubuehler of Zurich (CH)

Tijmen Verhulsdonck of Göteborg (SE)

Eloi Du Bois of Austin TX (US)

William Welch of San Francisco CA (US)

Charles Shang of San Mateo CA (US)

Ian Sachs of Corte Madera CA (US)

Kiran Bhat of San Francisco CA (US)

ROBUST FACIAL ANIMATION FROM VIDEO AND AUDIO

This abstract first appeared for US patent application 20240428492 titled 'ROBUST FACIAL ANIMATION FROM VIDEO AND AUDIO

Original Abstract Submitted

implementations described herein relate to methods, systems, and computer-readable media to generate animations for a 3d avatar from input video and audio captured at a client device. a camera may capture video of a face while a trained face detection model and a trained regression model output a set of video facs weights, head poses, and facial landmarks to be translated into the animations of the 3d avatar. additionally, a microphone may capture audio uttered by a user while a trained facial movement detection model and a trained regression model output a set of audio facs weights. additionally, a blending term is provided for identification of lapses in audio. a modularity mixing component fuses the video facs weights and the audio facs weights based on the blending term to create final facs weights for animating the user's avatar, a character rig, or another animation-capable construct.