18814438. ASYNCHRONOUS NEURAL NETWORK TRAINING (Microsoft Technology Licensing, LLC)

ASYNCHRONOUS NEURAL NETWORK TRAINING

Organization Name

Microsoft Technology Licensing, LLC

Inventor(s)

Ryota Tomioka of Cambridge (GB)

Matthew Alastair Johnson of Cambridge (GB)

Daniel Stefan Tarlow of Cambridge (GB)

Samuel Alexander Webster of Cambridge (GB)

Dimitrios Vytiniotis of Cambridge (GB)

Alexander Lloyd Gaunt of Cambridge (GB)

Maik Riechert of Cambridge (GB)

ASYNCHRONOUS NEURAL NETWORK TRAINING

This abstract first appeared for US patent application 18814438 titled 'ASYNCHRONOUS NEURAL NETWORK TRAINING

Original Abstract Submitted

A neural network training apparatus is described which has a network of worker nodes each having a memory storing a subgraph of a neural network to be trained. The apparatus has a control node connected to the network of worker nodes. The control node is configured to send training data instances into the network to trigger parallelized message passing operations which implement a training algorithm which trains the neural network. At least some of the message passing operations asynchronously update parameters of individual subgraphs of the neural network at the individual worker nodes.