18105159. EXPLOITING ACTIVATION SPARSITY IN DEEP NEURAL NETWORKS simplified abstract (QUALCOMM Incorporated)
Contents
EXPLOITING ACTIVATION SPARSITY IN DEEP NEURAL NETWORKS
Organization Name
Inventor(s)
Rexford Alan Hill of San Diego CA (US)
Aaron Douglass Lamb of San Diego CA (US)
Michael Goldfarb of San Diego CA (US)
Amin Ansari of Federal Way WA (US)
Christopher Lott of San Diego CA (US)
EXPLOITING ACTIVATION SPARSITY IN DEEP NEURAL NETWORKS - A simplified explanation of the abstract
This abstract first appeared for US patent application 18105159 titled 'EXPLOITING ACTIVATION SPARSITY IN DEEP NEURAL NETWORKS
Simplified Explanation
The abstract describes a method for exploiting activation sparsity in deep neural networks. Here is a simplified explanation of the abstract:
- The method involves retrieving two tensors: an activation tensor and a weight tensor.
- The activation tensor is a sparse tensor, meaning it contains mostly zero values.
- A compressed activation tensor is generated from the activation tensor, which only includes the non-zero activations.
- The compressed activation tensor has fewer columns than the original activation tensor.
- The compressed activation tensor and the weight tensor are then processed together to generate an output tensor.
Potential Applications:
- This method can be applied in various fields that utilize deep neural networks, such as computer vision, natural language processing, and speech recognition.
- It can improve the efficiency and speed of neural network computations, making it useful in real-time applications like autonomous vehicles, robotics, and video processing.
Problems Solved:
- Activation sparsity is a common characteristic in deep neural networks, where most activations are zero.
- This method addresses the challenge of efficiently processing sparse activation tensors, reducing computational requirements and memory usage.
Benefits:
- By exploiting activation sparsity, the method reduces the size and complexity of computations in deep neural networks.
- It improves the efficiency and speed of neural network operations, enabling faster inference and training.
- The method can lead to more energy-efficient implementations of deep neural networks, making them suitable for resource-constrained devices like smartphones and IoT devices.
Original Abstract Submitted
A method of exploiting activation sparsity in deep neural networks is described. The method includes retrieving an activation tensor and a weight tensor where the activation tensor is a sparse activation tensor. The method also includes generating a compressed activation tensor comprising non-zero activations of the activation tensor, where the compressed activation tensor has fewer columns than the activation tensor. The method further includes processing the compressed activation tensor and the weight tensor to generate an output tensor.
