18147297. NEURAL NETWORK DISTILLATION METHOD AND APPARATUS simplified abstract (HUAWEI TECHNOLOGIES CO., LTD.)
NEURAL NETWORK DISTILLATION METHOD AND APPARATUS
Organization Name
Inventor(s)
NEURAL NETWORK DISTILLATION METHOD AND APPARATUS - A simplified explanation of the abstract
This abstract first appeared for US patent application 18147297 titled 'NEURAL NETWORK DISTILLATION METHOD AND APPARATUS
Simplified Explanation
The patent application describes a method for distilling knowledge from one neural network to another in the field of artificial intelligence. This involves using two neural networks to process data and obtain target outputs through kernel function-based transformations. The first target output is obtained from the output of the first neural network layer, and the second target output is obtained from the output of the second neural network layer. Knowledge distillation is then performed on the first neural network using a target loss constructed from the first and second target outputs.
- The method involves using two neural networks to process data and obtain target outputs.
- The first target output is obtained through a kernel function-based transformation on the output of the first neural network layer.
- The second target output is obtained through a kernel function-based transformation on the output of the second neural network layer.
- Knowledge distillation is performed on the first neural network using a target loss constructed from the first and second target outputs.
Potential Applications
This technology has potential applications in various fields, including:
- Artificial intelligence research and development
- Machine learning algorithms
- Neural network training and optimization
- Data analysis and pattern recognition
Problems Solved
The technology addresses the following problems:
- Knowledge transfer between neural networks
- Improving the performance and efficiency of neural networks
- Enhancing the accuracy and reliability of artificial intelligence systems
- Overcoming limitations in training large neural networks
Benefits
The technology offers the following benefits:
- Improved knowledge transfer and distillation between neural networks
- Enhanced performance and efficiency of neural network models
- Increased accuracy and reliability of artificial intelligence systems
- Potential for faster and more effective training of large neural networks
Original Abstract Submitted
The technology of this application relates to a neural network distillation method, applied to the field of artificial intelligence, and includes processing to-be-processed data by using a first neural network and a second neural network to obtain a first target output and a second target output, where the first target output is obtained by performing kernel function-based transformation on an output of the first neural network layer, and the second target output is obtained by performing kernel function-based transformation on an output of the second neural network layer. The method further includes performing knowledge distillation on the first neural network based on a target loss constructed by using the first target output and the second target output.
