MULTI-MODAL MIXTURE OF EXPERTS NEURAL NETWORKS

Abstract: methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a multi-modal machine learning task using a neural network. in one aspect, a method comprises, receiving a request to perform a machine learning task on an input tuple comprising a first network input in a first modality and a second network input in a second modality; processing the first network input to generate a first embedded sequence; processing the second network input to generate a second embedded sequence; processing the first embedded sequence and the second embedded sequence using an attention neural network to generate an updated first embedded sequence and an updated second embedded sequence; and processing the updated first embedded sequence and the updated second embedded sequence to generate a final representation for the first network input and a final representation for the second network input.

Inventor(s): Basil Mustafa, Carlos Riquelme Ruiz, Joan Puigcerver i Perez, Rodolphe Jenatton, Neil Matthew Tinmouth Houlsby

CPC Classification: G06N3/08 (Learning methods)

Search for rejections for patent application number 20250217641