Abstract: Apparatus and methods for training neural networks based on a performance metric, including adjusting numerical precision and topology as training progresses are disclosed. In some examples, block floating-point formats having relatively lower accuracy are used during early stages of training. Accuracy of the floating-point format can be increased as training progresses based on a determined performance metric. In some examples, values for the neural network are transformed to normal precision floating-point formats. The performance metric can be determined based on entropy of values for the neural network, accuracy of the neural network, or by other suitable techniques. Accelerator hardware can be used to implement certain implementations, including hardware having direct support for block floating-point formats.

Abstract: A machine learning tool uses dithered quantization of parameters during training of a machine learning model such as a neural network. The machine learning tool receives training data and initializes certain parameters of the machine learning model (e.g., weights for connections between nodes of a neural network, biases for nodes). The machine learning tool trains the parameters in one or more iterations based on the training data. In particular, in a given iteration, the machine learning tool applies the machine learning model to at least some of the training data and, based at least in part on the results, determines parameter updates to the parameters. The machine learning tool updates the parameters using the parameter updates and a dithered quantizer function, which can add random values before a rounding or truncation operation.

Abstract: Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a narrower numerical precision than the first block floating-point format. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Abstract: Methods and apparatus are disclosed for providing emulation of quantized precision operations in a neural network. In some examples, the quantized precision operations are performed in a block floating-point format where values of a tensor share a common exponent. Techniques for selecting higher precision or lower precision can be used based on a variety of input metrics. When converting to a quantized tensor, a residual tensor is produced. In one embodiment, an error value associated with converting from a normal-precision floating point number to the quantized tensor is used to determine whether to use the residual tensor in a dot product calculation. Using the residual tensor increases the precision of an output from a node. Selection of whether to use the residual tensor can depend on various input metrics including the error value, the layer number, the exponent value, the layer type, etc.

Abstract: The Internet generally provides anonymity to the online activities of visitors to websites and other online resources. This prevents the operators of websites and others from identifying visitors who do not wish to be identified. Accordingly, embodiments generate mappings between entities (e.g., IP addresses, domains, cookies, or devices) and accounts (e.g., companies) to de-anonymize online activities. In an embodiment, summary mappings are generated based on activity data. Each summary mapping may comprise an entity, potential account identifier, and an activity vector that measures observations of an association between the entity and potential account identifier from an activity source for multiple summary periods. A model may be applied to the summary mappings to compute signal strengths for a plurality of candidate mappings. A winning mapping may then be selected for each entity in the candidate mappings, and used to associate the entity with an account in one or more downstream functions.

Abstract: Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a narrower numerical precision than the first block floating-point format. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Abstract: Methods and systems are provided for decoding variable-length codes in a parallel process. A stream of variable-length code words is divided into fixed length words. A plurality of parallel sets of decoder circuits each receive, in parallel, a current fixed length word and a prior fixed length word. Each decoder circuit has a respective fixed leftover bit-count. Each decoder circuit generates a respective output that may include a decoded symbol and a new leftover bit-count. Each respective output is determined based on the respective current fixed length word, the respective prior fixed length word, and the respective fixed leftover bit-count. A set of selected decoder circuit outputs is generated for each set of the parallel sets of decoder circuits based on a set of first leftover bit-counts. One output from each set of selected decoder circuit outputs is selected as a final output based on a second prior leftover bit-count.

Abstract: The Internet generally provides anonymity to the online activities of visitors to web sites and other online resources. This prevents the operators of web sites and others from identifying visitors who do not wish to be identified. Accordingly, embodiments generate mappings between entities (e.g., IP addresses, domains, cookies, or devices) and accounts (e.g., companies) to de-anonymize online activities. In an embodiment, summary mappings are generated based on activity data. Each summary mapping may comprise an entity, potential account identifier, and an activity vector that measures observations of an association between the entity and potential account identifier from an activity source for multiple summary periods. A model may be applied to the summary mappings to compute signal strengths for a plurality of candidate mappings. A winning mapping may then be selected for each entity in the candidate mappings, and used to associate the entity with an account in one or more downstream functions.

Abstract: Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissas for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a non-uniform and/or lossy mantissa. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Abstract: Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and, in particular, for adjusting floating-point formats used to store activation values during training. In certain examples of the disclosed technology, a computing system includes processors, memory, and a floating-point compressor in communication with the memory. The computing system is configured to produce a neural network comprising activation values expressed in a first floating-point format, select a second floating-point format for the neural network based on a performance metric, convert at least one of the activation values to the second floating-point format, and store the compressed activation values in the memory. Aspects of the second floating-point format that can be adjusted include the number of bits used to express mantissas, exponent format, use of non-uniform mantissas, and/or use of outlier values to express some of the mantissas.

Abstract: Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and, in particular, for adjusting floating-point formats used to store activation values during training. In certain examples of the disclosed technology, a computing system includes processors, memory, and a floating-point compressor in communication with the memory. The computing system is configured to produce a neural network comprising activation values expressed in a first floating-point format, select a second floating-point format for the neural network based on a performance metric, convert at least one of the activation values to the second floating-point format, and store the compressed activation values in the memory. Aspects of the second floating-point format that can be adjusted include the number of bits used to express mantissas, exponent format, use of non-uniform mantissas, and/or use of outlier values to express some of the mantissas.

Abstract: Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissas for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a non-uniform and/or lossy mantissa. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Abstract: The performance of a text parser implemented with a state machine is improved by reducing a critical dependence path. In one aspect, all possible current states for a given text input are read from a state table circuit, and the correct next state and output are then selected therefrom by an output multiplexer based on the current state, removing dependence on the current state from the table read, and allowing the read(s) to be pipelined. Further, multiple input units are configured to operate on multiple text characters in parallel, with each input unit propagating outputs for its state table circuit to the next downstream input unit. Each downstream input unit is configured to use the propagated states to provide the proper outputs to appropriates multiplexer inputs. The number of possible output states may be dynamically reduced, thereby reducing the size of the output multiplexer needed to select the next state.

Abstract: Apparatus and methods for training a neural network accelerator using quantized precision data formats having outlier values are disclosed, and in particular for storing activation values from a neural network in a compressed format for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a narrower numerical precision than the first block floating-point format. Outlier values, comprising additional bits of mantissa and/or exponent are stored in ancillary storage for subset of the activation values. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Abstract: The performance of a text parser implemented with a state machine is improved by reducing a critical dependence path. In one aspect, all possible current states for a given text input are read from a state table circuit, and the correct next state and output are then selected therefrom by an output multiplexer based on the current state, removing dependence on the current state from the table read, and allowing the read(s) to be pipelined. Further, multiple input units are configured to operate on multiple text characters in parallel, with each input unit propagating outputs for its state table circuit to the next downstream input unit. Each downstream input unit is configured to use the propagated states to provide the proper outputs to appropriates multiplexer inputs. The number of possible output states may be dynamically reduced, thereby reducing the size of the output multiplexer needed to select the next state.

Abstract: Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissas for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a non-uniform and/or lossy mantissa. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Abstract: A table-based state machine is improved by reducing critical dependence path. In one aspect, all current states for a given input are read from a state table circuit, and the next state and output are then selected therefrom by an output multiplexer based on the current state, removing dependence on the current state from the table read, and allowing the read(s) to be pipelined. In a further aspect, multiple input units are configured to operate on multiple inputs in parallel, with each input unit propagating its state table circuit for its current input to the next downstream input unit. Each downstream input unit is configured to use the propagated state table circuit to provide the state table circuit reads to the proper output multiplexer input. The number of possible output states for a given input may be dynamically reduced, reducing the size of the output multiplexer selecting the next state.

Abstract: Bounding box quantization can reduce the quantity of bits utilized to express numerical values prior to the multiplication of matrices comprised of such numerical values, thereby reducing both memory consumption and processor utilization. Stochastic rounding can provide sufficient precision to enable the storage of weight values in reduced-precision formats without having to separately store weight values in a full-precision format. Alternatively, other rounding mechanisms, such as round to nearest, can be utilized to exchange weight values in reduced-precision formats, while also storing weight values in full-precision formats for subsequent updating. To facilitate conversion, reduced-precision formats such as brain floating-point format can be utilized.

Abstract: Methods and systems are provided for decoding variable-length codes in a parallel process. A stream of variable-length code words is divided into fixed length words. A plurality of parallel sets of decoder circuits each receive, in parallel, a current fixed length word and a prior fixed length word. Each decoder circuit has a respective fixed leftover bit-count. Each decoder circuit generates a respective output that may include a decoded symbol and a new leftover bit-count. Each respective output is determined based on the respective current fixed length word, the respective prior fixed length word, and the respective fixed leftover bit-count. A set of selected decoder circuit outputs is generated for each set of the parallel sets of decoder circuits based on a set of first leftover bit-counts. One output from each set of selected decoder circuit outputs is selected as a final output based on a second prior leftover bit-count.

Abstract: A system for training a neural network receives training data and performing lower precision format training calculations using lower precision format data at one or more training phases. One or more results from the lower precision format training calculations are converted to higher precision format data, and higher precision format training calculations are performed using the higher precision format data at one or more additional training phases. The neural network is modified using the results from the one or more additional training phases. The mixed precision format training calculations train the neural network more efficiently, while maintaining an overall accuracy.