Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training giant neural networks. One of the methods includes obtaining data specifying a partitioning of the neural network into N composite layers that form a sequence of composite layers, wherein each composite layer comprises a distinct plurality of layers from the multiple network layers of the neural network; obtaining data assigning each of the N composite layers to one or more computing devices from a set of N computing devices; partitioning a mini-batch of training examples into a plurality of micro-batches; and training the neural network, comprising: performing a forward pass through the neural network until output activations have been computed for each micro-batch for a final composite layer in the sequence, and performing a backward pass through the neural network until output gradients have been computed for each micro-batch for the first composite layer in the sequence.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer. Some or all of the attention layers have a feed-forward sub-layer that applies conditional computation to the inputs to the sub-layer.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for feedback-directed optimization. One of the methods includes maintaining a data store comprising a plurality of optimization profiles that are used by a compiler to compile respective computer programs. The computer programs can be invoked by a set of executing workloads. Operations are repeatedly performed that include, for each optimization profile in at least a subset of the optimization profiles: determining or predicting whether the optimization profile is a valid optimization profile for a current software version of the compiler, and in response to determining or predicting that the optimization profile is not a valid optimization profile for the current software version of the compiler, removing the optimization profile from the data store.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer. Some or all of the attention layers have a feed-forward sub-layer that applies conditional computation to the inputs to the sub-layer.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training giant neural networks. One of the methods includes obtaining data specifying a partitioning of the neural network into N composite layers that form a sequence of composite layers, wherein each composite layer comprises a distinct plurality of layers from the multiple network layers of the neural network; obtaining data assigning each of the N composite layers to one or more computing devices from a set of N computing devices; partitioning a mini-batch of training examples into a plurality of micro-batches; and training the neural network, comprising: performing a forward pass through the neural network until output activations have been computed for each micro-batch for a final composite layer in the sequence, and performing a backward pass through the neural network until output gradients have been computed for each micro-batch for the first composite layer in the sequence.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training giant neural networks. One of the methods includes obtaining data specifying a partitioning of the neural network into N composite layers that form a sequence of composite layers, wherein each composite layer comprises a distinct plurality of layers from the multiple network layers of the neural network; obtaining data assigning each of the N composite layers to one or more computing devices from a set of N computing devices; partitioning a mini-batch of training examples into a plurality of micro-batches; and training the neural network, comprising: performing a forward pass through the neural network until output activations have been computed for each micro-batch for a final composite layer in the sequence, and performing a backward pass through the neural network until output gradients have been computed for each micro-batch for the first composite layer in the sequence.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training giant neural networks. One of the methods includes obtaining data specifying a partitioning of the neural network into N composite layers that form a sequence of composite layers, wherein each composite layer comprises a distinct plurality of layers from the multiple network layers of the neural network; obtaining data assigning each of the N composite layers to one or more computing devices from a set of N computing devices; partitioning a mini-batch of training examples into a plurality of micro-batches; and training the neural network, comprising: performing a forward pass through the neural network until output activations have been computed for each micro-batch for a final composite layer in the sequence, and performing a backward pass through the neural network until output gradients have been computed for each micro-batch for the first composite layer in the sequence.

Abstract: A system and method for improving the performance of all applications are disclosed. Production profile data may be collected about each application while the application is executing. The production profile data may be converted into symbolized profiles and stored in a database. The symbolized profiles may be aggregated into a single aggregated profile. This aggregated profile may be used as a compilation input when compiling new versions of an application's binary to improve the application's performance for observed application behavior.

Abstract: A system and method for using inline stacks to improve the performance of application binaries is included. While executing a first application binary, profile data may be collected about the application that includes which callee functions are called from the application's callsites and the number of times each inline stack is executed. A context summary map may be created from the collected profile data which shows a summary of the total execution count of all instructions in the callee function for each callsite inlined in the application's normal binary. Using the context summary map, each function callsite's execution count may be compared with a predetermined threshold to determine if the function should be inlined. Then the application's profile may be annotated and a second application binary, an optimized binary, may be generated using the annotated profile.

Abstract: While optimizing executable code, compilers traditionally make static determinations about whether or not to inline functions. Embodiments of the invention convert dynamic hardware-event sampling information into context-specific edge frequencies, which can be used to make inlining decisions for functions.

Abstract: Traditional feedback-directed optimization (FDO) is not widely used due to the significant computational overhead involved in using instrumented binaries. The described embodiments provide methods that eliminate the need for instrumented binaries by permitting the conversion of hardware-event sampling information into edge frequencies usable by FDO compilers. Some advantages include: the ability to collect feedback data on production systems; the ability to perform FDO on the OS kernel; and the ability to avoid disrupting timing paths from instrumented binaries.