Google llc (20250005924). Visual Transformers with Sparse Application of Video Kernels
Contents
Visual Transformers with Sparse Application of Video Kernels
Organization Name
Inventor(s)
Anthony J. Piergiovanni of Denver CO US
Wei-Cheng Kuo of Santa Clara CA US
Anelia Angelova of Palo Alto CA US
Visual Transformers with Sparse Application of Video Kernels
This abstract first appeared for US patent application 20250005924 titled 'Visual Transformers with Sparse Application of Video Kernels
Original Abstract Submitted
provided are machine-learned models for performing video processing with improved efficiency. in particular, the machine-learned model can perform the sparse application of one or more video kernels to a set of video data to generate video tokens that can, for example, be provided as input to a visual transformer. thus, example implementations of the present disclosure are directed to an approach which can turn a visual transformer (e.g., a vit encoder) into an efficient video model. furthermore, example implementations described herein can seamlessly work with both image and video inputs. specifically, by sparsely sampling the inputs, the model is able to do training and inference from both inputs. the proposed model is easily scalable and can optionally be adapted to large-scale pre-trained visual transformers without requiring full finetuning.
