View source for US Patent Application 18365047. NORMALIZATION SCHEME FOR SELF-ATTENTION NEURAL NETWORKS simplified abstract

=NORMALIZATION SCHEME FOR SELF-ATTENTION NEURAL NETWORKS=


==Organization Name==

[[:Category:Huawei Technologies Co., Ltd.|Huawei Technologies Co., Ltd.]]


[[Category:Huawei Technologies Co., Ltd.]]==Inventor(s)==

[[Category:[[:Category:Aladin Virmaux of Boulogne Billancourt (FR)|Aladin Virmaux of Boulogne Billancourt (FR)]]]]

[[Category:[[:Category:George Dasoulas of Boulogne Billancourt (FR)|George Dasoulas of Boulogne Billancourt (FR)]]]]

[[Category:[[:Category:Kevin Scaman of Munich (DE)|Kevin Scaman of Munich (DE)]]]]

==NORMALIZATION SCHEME FOR SELF-ATTENTION NEURAL NETWORKS - A simplified explanation of the abstract==


This abstract first appeared for US patent application 18365047 titled 'NORMALIZATION SCHEME FOR SELF-ATTENTION NEURAL NETWORKS

==Simplified Explanation==

The patent application describes a data processing device that performs attention-based operations on a graph neural network.
* The device receives one or more input graphs, each with multiple nodes.
* For each input graph, the device forms an input node representation for each node, with a defined norm for each representation.
* The device forms a set of attention parameters.
* The input node representations are multiplied with the attention parameters to form a score function for the input graph.
* The score function is normalized based on the maximum norms of the input node representations, resulting in a normalized score function.
* A weighted node representation is formed by weighting each node in the input graph using an element of the normalized score function.
* The normalization of the score function improves the performance of deep attention-based neural networks by enforcing Lipschitz continuity.


==Original Abstract Submitted==

Described is a data processing device for performing an attention-based operation on a graph neural network. The device is configured to receive one or more input graphs each having a plurality of nodes and to, for at least one of the input graphs: form an input node representation for each node in the respective input graph, wherein a respective norm is defined for each input node representation; form a set of attention parameters; multiply each of the input node representations with each of the set of attention parameters to form a score function of the respective input graph; normalize the score function based on a maximum of the norms of the input node representations to form a normalised score function; and form a weighted node representation by weighting each node in the respective input graph using a respective element of the normalised score function. The normalization of the score function enables deep attention-based neural networks to perform better by enforcing Lipschitz continuity.



[[Category:G06N3/048]]