17549090. EXTRACTING EXPLANATIONS FROM ATTENTION-BASED MODELS simplified abstract (INTERNATIONAL BUSINESS MACHINES CORPORATION)

EXTRACTING EXPLANATIONS FROM ATTENTION-BASED MODELS

Organization Name

INTERNATIONAL BUSINESS MACHINES CORPORATION

Inventor(s)

Thai F. Le of West Palm Beach FL (US)

Supriyo Chakraborty of White Plains NY (US)

Mudhakar Srivatsa of White Plains NY (US)

EXTRACTING EXPLANATIONS FROM ATTENTION-BASED MODELS - A simplified explanation of the abstract

This abstract first appeared for US patent application 17549090 titled 'EXTRACTING EXPLANATIONS FROM ATTENTION-BASED MODELS

Abstract

The patent application describes a method for explaining the outcome of a model by using an attention-based neural network. The network learns attention weights for tokens in the input data and predicts an outcome based on these weights. The method calculates a signed relevance score for each token, which quantifies its relevance to the outcome. This score is used to provide an explanation of the token's contribution to or against the outcome. The score is computed as a gradient of loss with respect to the attention weight.

Explanation of the Patent/Innovation

• The patent application proposes a method for explaining the outcome of a model using an attention-based neural network.
• The network learns attention weights for tokens in the input data, which helps in predicting the outcome.
• A signed relevance score is calculated for each token, quantifying its relevance to the outcome.
• This score is used to provide an explanation of the token's contribution to or against the outcome.
• The score is computed as a gradient of loss with respect to the attention weight, allowing for a precise calculation.

Potential Applications

• This technology can be applied in various fields where model outcomes need to be explained, such as finance, healthcare, and natural language processing.
• It can be used to explain the decisions made by machine learning models, providing transparency and accountability.
• The method can be integrated into existing AI systems to enhance their interpretability and trustworthiness.

Problems Solved by this Technology

• One of the main problems solved by this technology is the lack of interpretability in machine learning models.
• By providing explanations for model outcomes, this method helps in understanding the reasoning behind the decisions made by the model.
• It addresses the black-box nature of many AI systems, allowing users to trust and verify the results.

Benefits of this Technology

• The method allows for a more transparent and interpretable AI system, which can be crucial in domains where decision-making needs to be explainable.
• It provides insights into the factors influencing the model's outcome, helping users identify potential biases or errors.
• The technology can improve the trustworthiness of AI systems, as users can understand and validate the reasoning behind the decisions made by the model.

Original Abstract Submitted

Providing an explanation for model outcome can include receiving input data, and passing the input data through an attention-based neural network, where the attention-based neural network learns attention weights associated with contextual embeddings corresponding to tokens of the input data and predicts an outcome corresponding to the input data. Based on an attention weight associated with a contextual embedding corresponding to a token of the input data, a signed relevance score can be determined to associate with the token for quantifying the token's relevance to the outcome. Based on the signed relevance score, an explanation of the token's contribution toward or against the outcome can be provided. The signed relevance score can be computed as a gradient of loss with respect to the attention weight.