GOOGLE LLC (20240403362). Video and Audio Multimodal Searching System

Video and Audio Multimodal Searching System

Organization Name

GOOGLE LLC

Inventor(s)

Harshit Kharbanda of Pleasanton CA (US)

Belinda Luna Zeng of Cupertino CA (US)

Viviana Caso Corella of San Francisco CA (US)

Aashi Jain of Sunnyvale CA (US)

David William Hendon of Oakland CA (US)

Christopher James Kelley of Orinda CA (US)

Jessica Lee of Brooklyn NY (US)

Dounia Berrada of Saratoga CA (US)

Kai Yu of San Francisco CA (US)

Louis Wang of San Francisco CA (US)

Thomas J. Duerig of Mountain View CA (US)

Radu Soricut of Manhattan Beach CA (US)

Robin Dua of San Francisco CA (US)

Video and Audio Multimodal Searching System

This abstract first appeared for US patent application 20240403362 titled 'Video and Audio Multimodal Searching System

Original Abstract Submitted

a multimodal search system using a video query is described. the system can receive video data captured by a camera of a user device. the video data can have a sequence of image frames. additionally, the system can receive audio data associated with the video data captured by the user device. moreover, the system can process, using one or more machine-learned models, the sequence of image frames to generate video embeddings related to the sequence of the image frames. the video embeddings can have a plurality of image embeddings associated with the sequence of image frames. furthermore, the system can determine one or more video results based on the video embeddings and the audio data. subsequently, the system can transmit, to the user device, the one or more video results.