18306881. SYSTEM AND METHOD FOR USING RESIDUAL TRANSFORMERS IN NATURAL LANGUAGE PROCESSING simplified abstract (Samsung Electronics Co., Ltd.)

From WikiPatents
Jump to navigation Jump to search

SYSTEM AND METHOD FOR USING RESIDUAL TRANSFORMERS IN NATURAL LANGUAGE PROCESSING

Organization Name

Samsung Electronics Co., Ltd.

Inventor(s)

Sai Ajay Modukuri of San Francisco CA (US)

Brendon Christopher Beachy Eby of Chicago IL (US)

Suhel Jaber of San Jose CA (US)

SYSTEM AND METHOD FOR USING RESIDUAL TRANSFORMERS IN NATURAL LANGUAGE PROCESSING - A simplified explanation of the abstract

This abstract first appeared for US patent application 18306881 titled 'SYSTEM AND METHOD FOR USING RESIDUAL TRANSFORMERS IN NATURAL LANGUAGE PROCESSING

Simplified Explanation

The method described in the patent application involves using embedding vectors to represent tokens in an input to a transformer. The transformer consists of multiple transformer layers arranged in a sequence, with each layer having a residual connection to the previous layer.

For each transformer layer, the method determines an input embedding vector for the first token based on a combination of output embedding vectors from previous transformer layers. It then processes this input embedding vector to generate an output embedding vector, which is provided to each subsequent transformer layer.

  • The method uses embedding vectors to represent tokens in an input.
  • The transformer consists of multiple transformer layers arranged in a sequence.
  • Each transformer layer has a residual connection to the previous layer.
  • For each transformer layer, an input embedding vector is determined for the first token based on output embedding vectors from previous layers.
  • The input embedding vector is processed to generate an output embedding vector.
  • The output embedding vector is provided to each subsequent transformer layer.

Potential Applications:

  • Natural language processing tasks such as language translation, sentiment analysis, and text generation.
  • Speech recognition and synthesis.
  • Image and video captioning.
  • Chatbots and virtual assistants.
  • Recommendation systems.

Problems Solved:

  • The method improves the representation of tokens in a transformer model by using embedding vectors.
  • It allows for better understanding and processing of natural language inputs.
  • It helps in capturing contextual information and dependencies between tokens.

Benefits:

  • Enhanced performance in various natural language processing tasks.
  • Improved accuracy and efficiency in language translation, sentiment analysis, and other applications.
  • Better understanding and interpretation of complex inputs.
  • Enables the development of more advanced and sophisticated AI models.


Original Abstract Submitted

A method includes providing embedding vectors representing tokens in an input to a transformer comprising multiple transformer layers arranged in a sequence, each transformer layer having a residual connection to each previous transformer layer. The method also includes, for each transformer layer, determining, for a first token, an input embedding vector based on a combination of output embedding vectors from previous transformer layers. The method further includes, for each transformer layer, processing, for the first token, the input embedding vector to generate an output embedding vector to be provided to each subsequent transformer layer.

Retrieved from "http://wikipatents.org/index.php?title=18306881._SYSTEM_AND_METHOD_FOR_USING_RESIDUAL_TRANSFORMERS_IN_NATURAL_LANGUAGE_PROCESSING_simplified_abstract_(Samsung_Electronics_Co.,_Ltd.)&oldid=28954"