18489503. Efficient Training Mixture Calibration for Training Machine-Learned Models (Google LLC)
Efficient Training Mixture Calibration for Training Machine-Learned Models
Organization Name
Inventor(s)
Sang Xie of Mountain View CA US
Hieu Hy Pham of Mountain View CA US
Efficient Training Mixture Calibration for Training Machine-Learned Models
This abstract first appeared for US patent application 18489503 titled 'Efficient Training Mixture Calibration for Training Machine-Learned Models
Original Abstract Submitted
Systems and methods are provided for efficiently calibrating a data mixture for training machine-learned models (e.g., machine-learned sequence processing models, such as transformer-based models). For example, machine-learned models can be trained over a broad dataset that can include multiple different categories of data. The mixture of data categories within the dataset can influence model performance. To improve the performance of machine-learned models, example implementations of the present disclosure can learn a distribution of data categories using a lightweight proxy model before initiating training of a large primary model. In this manner, for instance, example implementations can obtain an improved training data distribution with less computational expense and can leverage the learned training data distribution to better train a large primary model.
