Abstract: A system and method for iteratively updating a parameter according to a gradient descent algorithm. In a given nth iteration of the method, one or more processors may determine a gradient value of a gradient vector of the parameter in a first dimension, determine a product value based at least in part on a sum of (i) the product value determined in an n?1th iteration and (ii) a product of the determined gradient value and a learning rate of the gradient descent algorithm, determine an updated parameter value according to a function including the product value, and update the parameter to equal the updated parameter value.

Abstract: Methods, systems, and apparatus, including an apparatus for generating clusters of building blocks of compute nodes using an optical network. In one aspect, a method includes receiving data specifying requested compute nodes for a computing workload. The data specifies a target arrangement of the nodes. A subset of building blocks of a superpod is selected. A logical arrangement of the subset of compute nodes that matches the target arrangement is determined. A workload cluster of compute nodes that includes the subset of the building blocks is generated. For each dimension of the workload cluster, respective routing data for two or more OCS switches for the dimension is configured. One-to-many switches are configured such that a second compute node of each segment of compute nodes is connected to a same OCS switch as a corresponding first compute node of a corresponding segment to which the second compute node is connected.

Abstract: This specification describes systolic arrays of hardware processing units. In one aspect, a matrix computation unit includes multiple cells arranged in a systolic array. Each cell includes multiplication circuitry configured to determine a product of elements or submatrices of input matrices, summation circuitry configured to determine a sum of an input accumulated value and the product output by the multiplication circuitry, multiple accumulators connected to an output of the summation circuitry, and a controller circuit configured to select, from the accumulators, a given accumulator to receive the sum output by the summation circuitry.

Abstract: This specification relates to systolic arrays of hardware processing units. In one aspect, a matrix multiplication unit includes multiple cells arranged in a systolic array. Each cell includes multiplication circuitry configured to determine a product of elements of input matrices. Each cell includes an accumulator configured to determine an accumulated value by accumulating a sum of the products output by the multiplication circuitry. Each cell also includes a post-processing component configured to determine a post-processed value by performing one or more post-processing operations on the accumulated value.

Abstract: A system and method for iteratively updating a parameter according to a gradient descent algorithm. In a given nth iteration of the method, one or more processors may determine a gradient value of a gradient vector of the parameter in a first dimension, determine a product value based at least in part on a sum of (i) the product value determined in an n?1th iteration and (ii) a product of the determined gradient value and a learning rate of the gradient descent algorithm, determine an updated parameter value according to a function including the product value, and update the parameter to equal the updated parameter value.

Abstract: Methods, systems, and apparatus, including an apparatus for generating clusters of building blocks of compute nodes using an optical network. In one aspect, a method includes receiving data specifying requested compute nodes for a computing workload. The data specifies a target arrangement of the nodes. A subset of building blocks of a superpod is selected. A logical arrangement of the subset of compute nodes that matches the target arrangement is determined. A workload cluster of compute nodes that includes the subset of the building blocks is generated. For each dimension of the workload cluster, respective routing data for two or more OCS switches for the dimension is configured. One-to-many switches are configured such that a second compute node of each segment of compute nodes is connected to a same OCS switch as a corresponding first compute node of a corresponding segment to which the second compute node is connected.

Abstract: Methods, systems, and apparatus, including an apparatus for generating clusters of building blocks of compute nodes using an optical network. In one aspect, a method includes receiving data specifying requested compute nodes for a computing workload. The data specifies a target arrangement of the nodes. A subset of building blocks of a superpod is selected. A logical arrangement of the subset of compute nodes that matches the target arrangement is determined. A workload cluster of compute nodes that includes the subset of the building blocks is generated. For each dimension of the workload cluster, respective routing data for two or more OCS switches for the dimension is configured. One-to-many switches are configured such that a second compute node of each segment of compute nodes is connected to a same OCS switch as a corresponding first compute node of a corresponding segment to which the second compute node is connected.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using dynamic minibatch sizes during neural network training. One of the methods includes receiving, by each of a plurality of host computer, a respective batch of training examples, each training example having zero or more features, computing, by each host computer, a minimum number of minibatches into which the host computer can divide the respective batch of training examples so that the host computer can process each minibatch using an embedding layer of the neural network without exceeding available computing resources, determining a largest minimum number of minibatches (N) into which any host computer can divide its respective batch of training examples, generating, by each host computer, N minibatches from the respective batch of training examples received by the host computer, and processing, by each host computer, the N minibatches using the embedding layer.

Abstract: Methods, systems, and apparatus, including an apparatus for generating clusters of building blocks of compute nodes using an optical network. In one aspect, a method includes receiving data specifying requested compute nodes for a computing workload. The data specifies a target arrangement of the nodes. A subset of building blocks of a superpod is selected. A logical arrangement of the subset of compute nodes that matches the target arrangement is determined. A workload cluster of compute nodes that includes the subset of the building blocks is generated. For each dimension of the workload cluster, respective routing data for two or more OCS switches for the dimension is configured. One-to-many switches are configured such that a second compute node of each segment of compute nodes is connected to a same OCS switch as a corresponding first compute node of a corresponding segment to which the second compute node is connected.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using dynamic minibatch sizes during neural network training. One of the methods includes receiving, by each of a plurality of host computer, a respective batch of training examples, each training example having zero or more features, computing, by each host computer, a minimum number of minibatches into which the host computer can divide the respective batch of training examples so that the host computer can process each minibatch using an embedding layer of the neural network without exceeding available computing resources, determining a largest minimum number of minibatches (N) into which any host computer can divide its respective batch of training examples, generating, by each host computer, N minibatches from the respective batch of training examples received by the host computer, and processing, by each host computer, the N minibatches using the embedding layer.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using dynamic minibatch sizes during neural network training. One of the methods includes receiving, by each of a plurality of host computer, a respective batch of training examples, each training example having zero or more features, computing, by each host computer, a minimum number of minibatches into which the host computer can divide the respective batch of training examples so that the host computer can process each minibatch using an embedding layer of the neural network without exceeding available computing resources, determining a largest minimum number of minibatches (N) into which any host computer can divide its respective batch of training examples, generating, by each host computer, N minibatches from the respective batch of training examples received by the host computer, and processing, by each host computer, the N minibatches using the embedding layer.

Abstract: One embodiment is a method of operating a plenoptic image processing system. The method includes storing plenoptic image data in a non-transitory computer readable memory of the system. The plenoptic image data includes data of a plurality of microimages. The method includes applying a frequency domain transform to the stored plenoptic image data to provide a frequency domain plenoptic image data structure stored in the memory, and processing the data structure using an extended slice technique to select a focal plane from the data structure. The extended slice technique comprises taking multiple slices of the frequency domain transformed plenoptic image data structure and configuring the multiple slices to provide extended slice data. The method further includes applying an inverse frequency domain transform to the extended slice data to provide inverse transform extended slice data, and providing an image output based upon the inverse transformed extended slice data.

Abstract: One embodiment is a method of operating a plenoptic image processing system. The method includes storing plenoptic image data in a non-transitory computer readable memory of the system. The plenoptic image data includes data of a plurality of microimages. The method includes applying a frequency domain transform to the stored plenoptic image data to provide a frequency domain plenoptic image data structure stored in the memory, and processing the data structure using an extended slice technique to select a focal plane from the data structure. The extended slice technique comprises taking multiple slices of the frequency domain transformed plenoptic image data structure and configuring the multiple slices to provide extended slice data. The method further includes applying an inverse frequency domain transform to the extended slice data to provide inverse transform extended slice data, and providing an image output based upon the inverse transformed extended slice data.

Abstract: The present invention involves a sparse matrix processing system and method which uses sparse matrices that are compressed to reduce memory traffic and improve performance of computations using sparse matrices.

Abstract: The present invention involves a sparse matrix processing system and method which uses sparse matrices that are compressed to reduce memory traffic and improve performance of computations using sparse matrices.