17983130. Efficient Training of Embedding Models Using Negative Cache simplified abstract (Google LLC)
Efficient Training of Embedding Models Using Negative Cache
Organization Name
Inventor(s)
Erik Michael Lindgren of New York NY (US)
Sashank Jakkam Reddi of Jersey City NJ (US)
Sanjiv Kumar of Jericho NY (US)
Efficient Training of Embedding Models Using Negative Cache - A simplified explanation of the abstract
This abstract first appeared for US patent application 17983130 titled 'Efficient Training of Embedding Models Using Negative Cache
Simplified Explanation
The patent application describes systems and methods for more efficiently training embedding models using a cache of item embeddings.
- The cache contains "stale" embeddings generated by a previous version of the model at a previous training iteration.
- The cached item embeddings are used to generate similarity scores for sampling negative items in the current training iteration.
- Gumbel-Max sampling is used to sample negative items that approximate a true gradient.
- New embeddings are generated for the sampled negative items and used to train the model at the current iteration.
Potential Applications
- This technology can be applied in recommendation systems to improve the efficiency of training embedding models.
- It can be used in natural language processing tasks such as word embeddings to enhance training efficiency.
Problems Solved
- The technology addresses the problem of training embedding models efficiently by utilizing a cache of item embeddings.
- It solves the challenge of generating negative items for training iterations by using similarity scores and Gumbel-Max sampling.
Benefits
- The use of a cache of item embeddings reduces the computational cost of training embedding models.
- By generating new embeddings for negative items, the model can be trained more effectively.
- The technology improves the overall efficiency and performance of embedding models.
Original Abstract Submitted
Provided are systems and methods which more efficiency train embedding models through the use of a cache of item embeddings for candidate items over a number of training iterations. The cached item embeddings can be “stale” embeddings that were generated by a previous version of the model at a previous training iteration. Specifically, at each iteration, the (potentially stale) item embeddings included in the cache can be used when generating similarity scores that are the basis for sampling a number of items to use as negatives in the current training iteration. For example, a Gumbel-Max sampling approach can be used to sample negative items that will enable an approximation of a true gradient. New embeddings can be generated for the sampled negative items and can be used to train the model at the current iteration.
