Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the systems includes a sparse memory access subsystem that is configured to perform operations comprising generating a sparse set of reading weights that includes a respective reading weight for each of the plurality of locations in the external memory using the read key, reading data from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generating a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and writing the write vector to the plurality of locations in the external memory in accordance with the writing weights.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the systems includes a sparse memory access subsystem that is configured to perform operations comprising generating a sparse set of reading weights that includes a respective reading weight for each of the plurality of locations in the external memory using the read key, reading data from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generating a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and writing the write vector to the plurality of locations in the external memory in accordance with the writing weights.

Abstract: Apparatus and methods for developing parallel networks. Parallel network design may comprise a general purpose language (GPC) code portion and a network description (ND) portion. GPL tools may be utilized in designing the network. The GPL tools may be configured to produce network specification language (NSL) engine adapted to generate hardware optimized machine executable code corresponding to the network description. The developer may be enabled to describe a parameter of the network. The GPC portion may be automatically updated consistent with the network parameter value. The GPC byte code may be introspected by the NSL engine to provide the underlying source code that may be automatically reinterpreted to produce the hardware optimized machine code. The optimized machine code may be executed in parallel.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

Abstract: Apparatus and methods for developing parallel networks. Parallel network design may comprise a general purpose language (GPC) code portion and a network description (ND) portion. GPL tools may be utilized in designing the network. The GPL tools may be configured to produce network specification language (NSL) engine adapted to generate hardware optimized machine executable code corresponding to the network description. The developer may be enabled to describe a parameter of the network. The GPC portion may be automatically updated consistent with the network parameter value. The GPC byte code may be introspected by the NSL engine to provide the underlying source code that may be automatically reinterpreted to produce the hardware optimized machine code. The optimized machine code may be executed in parallel.

Abstract: Apparatus and methods for developing parallel networks. In some implementations, a network may be partitioned into multiple partitions, wherein individual portions are being executed by respective threads executed in parallel. Individual portions may comprise multiple neurons and synapses. In order to reduce cross-thread traffic and/or reduce number of synchronization locks, network may be partitioned such that for given network portion, the neurons and the input synapses into neurons within the portion are executed within the same thread. Synapse update rules may be configured to allow memory access for postsynaptic neurons and forbid memory access to presynaptic neurons. Individual threads may be afforded pairs of memory buffers configured to effectuate asynchronous data input/output to/from thread. During an even iteration of network operation, even buffer may be utilized to store data generated by the thread during even iteration.

Abstract: Apparatus and methods for developing parallel networks. Parallel network design may comprise a general purpose language (GPC) code portion and a network description (ND) portion. GPL tools may be utilized in designing the network. The GPL tools may be configured to produce network specification language (NSL) engine adapted to generate hardware optimized machine executable code corresponding to the network description. The developer may be enabled to describe a parameter of the network. The GPC portion may be automatically updated consistent with the network parameter value. The GPC byte code may be introspected by the NSL engine to provide the underlying source code that may be automatically reinterpreted to produce the hardware optimized machine code. The optimized machine code may be executed in parallel.

Abstract: Spiking neuron network conditionally independent subset classifier apparatus and methods. In some implementations, the network may comprise one or more subset neuron layers configured to determine presence of one or more features in the subset of plurality of conditionally independent features. The output of the subset layer may be coupled to an aggregation layer. State of the subset layer may be configured during training based on training input and a reference signal. During operation, spiking output of the subset layer may be combined by the aggregation layer to produce classifier output. Subset layer output and/or classifier output may be encoded using spike rate, latency, and/or base-n encoding.

Abstract: Apparatus and methods for developing parallel networks. Parallel network design may comprise a general purpose language (GPC) code portion and a network description (ND) portion. GPL tools may be utilized in designing the network. The GPL tools may be configured to produce network specification language (NSL) engine adapted to generate hardware optimized machine executable code corresponding to the network description. The developer may be enabled to describe a parameter of the network. The GPC portion may be automatically updated consistent with the network parameter value. The GPC byte code may be introspected by the NSL engine to provide the underlying source code that may be automatically reinterpreted to produce the hardware optimized machine code. The optimized machine code may be executed in parallel.