18613263. TRANSFORMER WITH MULTI-SCALE MULTI-CONTEXT ATTENTIONS (QUALCOMM Incorporated)

TRANSFORMER WITH MULTI-SCALE MULTI-CONTEXT ATTENTIONS

Organization Name

QUALCOMM Incorporated

Inventor(s)

Tianyu Jiang of San Diego CA (US)

Manish Kumar Singh of San Diego CA (US)

Hsin-Pai Cheng of San Diego CA (US)

Hong Cai of San Diego CA (US)

Mingu Lee of San Diego CA (US)

Kartikeya Bhardwaj of San Diego CA (US)

Christopher Lott of San Diego CA (US)

Fatih Murat Porikli of San Diego CA (US)

TRANSFORMER WITH MULTI-SCALE MULTI-CONTEXT ATTENTIONS

This abstract first appeared for US patent application 18613263 titled 'TRANSFORMER WITH MULTI-SCALE MULTI-CONTEXT ATTENTIONS

Original Abstract Submitted

Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. A transformed version of image pixels is accessed as input to an attention layer of a machine learning model. A number of local attention operations to apply, in one transformer, to the transformed version of image pixels is selected based at least in part on a size of the transformed version of image pixels. A transformer output for the attention layer of the machine learning model is generated based on applying the number of local attention operations and at least one global attention operation to the transformed version of image pixels.