TEXT CONDITIONED VIDEO RESAMPLER FOR VIDEO UNDERSTANDING

Abstract: methods, systems, and apparatus for video understanding. in one aspect, a conditioned resampler model receives video features of multiple video frames of a video processed by a visual encoder and token embeddings for a specified task. the conditioned resampler model generates conditioned resampler embeddings according to the specified task in response to the video features and token embeddings provided as input. the conditioned resampler embeddings are provided to a large language model as input. the large language model generates, in response to the input conditioned resampler embeddings, a text response to the specified task.

Inventor(s): Alessio Tonioni, Bruno Korbar, Federico Tombari, Andrew Zisserman, Yongqin Xian

CPC Classification: G06V20/46 (IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING)

Search for rejections for patent application number 20250166379