Google llc (20250124708). VIDEO-TEXT MODELING WITH ZERO-SHOT TRANSFER FROM CONTRASTIVE CAPTIONERS
VIDEO-TEXT MODELING WITH ZERO-SHOT TRANSFER FROM CONTRASTIVE CAPTIONERS
Organization Name
Inventor(s)
Zirui Wang of Mountain View CA US
Yuan Cao of Mountain View CA US
VIDEO-TEXT MODELING WITH ZERO-SHOT TRANSFER FROM CONTRASTIVE CAPTIONERS
This abstract first appeared for US patent application 20250124708 titled 'VIDEO-TEXT MODELING WITH ZERO-SHOT TRANSFER FROM CONTRASTIVE CAPTIONERS
Original Abstract Submitted
provided is an efficient approach to establish a foundational video-text model for tasks including open-vocabulary video classification, text-to-video retrieval, video captioning and video question-answering. some example implementations include a model which can be referred to as videococa. example implementations reuse a pretrained image-text contrastive captioner (coca) model and adapt it to video-text tasks with little or minimal extra training. while previous works adapt image-text models with various cross-frame fusion modules (for example, cross-frame attention layer or perceiver resampler) and finetune the modified architecture on video-text data, aspects of the present disclosure leverage findings that the generative attentional pooling and contrastive attentional pooling layers in the image-text coca design are instantly adaptable to âflattened frame embeddingsâ, yielding a strong zero-shot transfer baseline for many video-text tasks.