Google LLC (20250006217). Automatic Speech Recognition Accuracy With Multimodal Embeddings Search

Automatic Speech Recognition Accuracy With Multimodal Embeddings Search

Organization Name

Google LLC

Inventor(s)

Christopher Li of New York NY (US)

Kyle Scott Kastner of Waltham MA (US)

Yuan Wang of Hoboken NJ (US)

Zhehuai Chen of Edgewater NJ (US)

Andrew Maxwell Rosenberg of Brooklyn NY (US)

Heng Su of Beijing (CN)

Qian Chen of Beijing (CN)

Leonid Aleksandrovich Velikovich of New York NY (US)

Patrick Maxim Rondon of New York NY (US)

Diamantino Antonio Caseiro of Philadelphia PA (US)

Zelin Wu of Jersey City NJ (US)

Automatic Speech Recognition Accuracy With Multimodal Embeddings Search

This abstract first appeared for US patent application 20250006217 titled 'Automatic Speech Recognition Accuracy With Multimodal Embeddings Search

Original Abstract Submitted

a method includes receiving training data that includes a set of transcribed speech utterances where each respective transcribed speech utterance is paired with a corresponding transcription. for each respective transcribed speech utterance, the method includes generating an encoded audio representation and an encoded textual representation, generating a higher order audio feature representation for a corresponding encoded audio representation, generating a higher order textual feature representation for a corresponding encoded textual representation, and determining a loss for the respective transcribed speech utterance based on the higher order audio feature representation and the higher order textual feature representation. the method also includes training a speech encoder and a text encoder of a correction model based on the loss determined for each transcribed speech utterance of the set of transcribed speech utterances.