Abstract: Methods, systems and apparatus, including computer programs encoded on computer storage media, for training an action selection neural network. One of the methods includes receiving an observation characterizing a current state of the environment; determining a target network output for the observation by performing a look ahead search of possible future states of the environment starting from the current state until the environment reaches a possible future state that satisfies one or more termination criteria, wherein the look ahead search is guided by the neural network in accordance with current values of the network parameters; selecting an action to be performed by the agent in response to the observation using the target network output generated by performing the look ahead search; and storing, in an exploration history data store, the target network output in association with the observation for use in updating the current values of the network parameters.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for prediction of an outcome related to an environment. In one aspect, a system comprises a state representation neural network that is configured to: receive an observation characterizing a state of an environment being interacted with by an agent and process the observation to generate an internal state representation of the environment state; a prediction neural network that is configured to receive a current internal state representation of a current environment state and process the current internal state representation to generate a predicted subsequent state representation of a subsequent state of the environment and a predicted reward for the subsequent state; and a value prediction neural network that is configured to receive a current internal state representation of a current environment state and process the current internal state representation to generate a value prediction.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network having a plurality of policy parameters and used to select actions to be performed by an agent to control the agent to perform a particular task while interacting with one or more other agents in an environment. In one aspect, the method includes: maintaining data specifying a pool of candidate action selection policies; maintaining data specifying respective matchmaking policy; and training the policy neural network using a reinforcement learning technique to update the policy parameters. The policy parameters define policies to be used in controlling the agent to perform the particular task.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for asynchronous deep reinforcement learning. One of the systems includes a plurality of workers, wherein each worker is configured to operate independently of each other worker, and wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network.

Abstract: Systems and methods for facilitating updates to data pipelines using modularly-generated platform-agnostic data pipeline portions are disclosed. The system receives a user selection of a portion of a data pipeline comprising (i) a set of nodes each indicating a data pipeline component and (ii) a set of links linking the set of nodes. The system then generates a modular-portion of the data pipeline architecture, via a transformation component, based on the user selection and stores the modular-portion of the data pipeline architecture in a remote database. The system then receives an update to at least one node of the set of nodes of the modular-portion of the data pipeline architecture. The system then updates at least a subset of a set of pre-existing data pipelines that use the generated modular-portion of the data pipeline to incorporate the update to the at least one node of the set of nodes.

Abstract: Systems and methods for facilitating updates to data pipelines using modularly-generated platform-agnostic data pipeline portions are disclosed. The system receives a user selection of a portion of a data pipeline comprising (i) a set of nodes each indicating a data pipeline component and (ii) a set of links linking the set of nodes. The system then generates a modular-portion of the data pipeline architecture, via a transformation component, based on the user selection and stores the modular-portion of the data pipeline architecture in a remote database. The system then receives an update to at least one node of the set of nodes of the modular-portion of the data pipeline architecture. The system then updates at least a subset of a set of pre-existing data pipelines that use the generated modular-portion of the data pipeline to incorporate the update to the at least one node of the set of nodes.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network used to select actions performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes maintaining a replay memory, where the replay memory stores pieces of experience data generated as a result of the reinforcement learning agent interacting with the environment. Each piece of experience data is associated with a respective expected learning progress measure that is a measure of an expected amount of progress made in the training of the neural network if the neural network is trained on the piece of experience data. The method further includes selecting a piece of experience data from the replay memory by prioritizing for selection pieces of experience data having relatively higher expected learning progress measures and training the neural network on the selected piece of experience data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network having a plurality of policy parameters and used to select actions to be performed by an agent to control the agent to perform a particular task while interacting with one or more other agents in an environment. In one aspect, the method includes: maintaining data specifying a pool of candidate action selection policies; maintaining data specifying respective matchmaking policy; and training the policy neural network using a reinforcement learning technique to update the policy parameters. The policy parameters define policies to be used in controlling the agent to perform the particular task.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for asynchronous deep reinforcement learning. One of the systems includes a plurality of workers, wherein each worker is configured to operate independently of each other worker, and wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network.

Abstract: Methods, systems and apparatus, including computer programs encoded on computer storage media, for training an action selection neural network. One of the methods includes receiving an observation characterizing a current state of the environment; determining a target network output for the observation by performing a look ahead search of possible future states of the environment starting from the current state until the environment reaches a possible future state that satisfies one or more termination criteria, wherein the look ahead search is guided by the neural network in accordance with current values of the network parameters; selecting an action to be performed by the agent in response to the observation using the target network output generated by performing the look ahead search; and storing, in an exploration history data store, the target network output in association with the observation for use in updating the current values of the network parameters.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

Abstract: There are provided system and method of predicting data related to olfactory properties of a molecule characterized by a chemical structure. The method comprises: upon obtaining data informative of a spatial surface representation (SSR) of molecule corresponding to the chemical structure thereof, selecting on SSR a plurality of N surface points; for each selected surface point, obtaining local data informative of spatial location on SSR and local physicochemical properties of the selected surface point, thus giving rise to a surface points representation (SPR); inputting data informative of SPR into a Machine-Learned (ML) model trained to provide, in accordance with SPR, prediction data related to at least one olfactory property; and receiving, as an output of the ML model, prediction data related to the at least one olfactory property of the molecule. There are also provided system and method of predicting molecular chemical structure enabling one or more olfactory properties.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system. The method includes: training an action selection policy neural network, and during the training of the action selection neural network, training one or more auxiliary control neural networks and a reward prediction neural network. Each of the auxiliary control neural networks is configured to receive a respective intermediate output generated by the action selection policy neural network and generate a policy output for a corresponding auxiliary control task. The reward prediction neural network is configured to receive one or more intermediate outputs generated by the action selection policy neural network and generate a corresponding predicted reward.

Abstract: A method performed by one or more computers for obtaining an optimized algorithm that (i) is functionally equivalent to a target algorithm and (ii) optimizes one or more target properties when executed on a target set of one or more hardware devices. The method includes: initializing a target tensor representing the target algorithm; generating, using a neural network having a plurality of network parameters, a tensor decomposition of the target tensor that parametrizes a candidate algorithm; generating target property values for each of the target properties when executing the candidate algorithm on the target set of hardware devices; determining a benchmarking score for the tensor decomposition based on the target property values of the candidate algorithm; generating a training example from the tensor decomposition and the benchmarking score; and storing, in a training data store, the training example for use in updating the network parameters of the neural network.

Abstract: Aspects of the disclosure relate to generating scouting objectives in order to update map information used to control a fleet of vehicles in an autonomous driving mode. For instance, a notification from a vehicle of the fleet identifying a feature and a location of the feature may be received. A first bound for a scouting area may be identified based on the location of the feature. A second bound for the scouting area may be identified based on a lane closest to the feature. A scouting objective may be generated for the feature based on the first bound and the second bound.

Abstract: A computer-implemented method for generating an output token sequence from an input token sequence. The method combines a look ahead tree search, such as a Monte Carlo tree search, with a sequence-to-sequence neural network system. The sequence-to-sequence neural network system has a policy output defining a next token probability distribution, and may include a value neural network providing a value output to evaluate a sequence. An initial partial output sequence is extended using the look ahead tree search guided by the policy output and, in implementations, the value output, of the sequence-to-sequence neural network system until a complete output sequence is obtained.

Abstract: Aspects of the disclosure relate to detecting and responding to malfunctioning traffic signals for a vehicle having an autonomous driving mode. For instance, information identifying a detected state of a traffic signal for an intersection. An anomaly for the traffic signal may be detected based on the detected state and prestored information about expected states of the traffic signal. The vehicle may be controlled in the autonomous driving mode based on the detected anomaly.

Abstract: Imaging apparatus (22) includes a radiation source (40), which emits pulsed beams (42) of optical radiation toward a target scene (24). An array (52) of sensing elements (78) output signals indicative of respective times of incidence of photons in a first image of the target scene that is formed on the array of sensing elements. An image sensor (64) captures a second image of the target scene in registration with the first image. Processing and control circuitry (56, 58) identifies, responsively to the signals, areas of the array on which the pulses of optical radiation reflected from corresponding regions of the target scene are incident, and processes the signals from the sensing elements in the identified areas in order measure depth coordinates of the corresponding regions of the target scene based on the times of incidence, while identifying, responsively to the second image, one or more of the regions of the target scene as no-depth regions.

Abstract: Aspects of the disclosure relate to generating scouting objectives in order to update map information used to control a fleet of vehicles in an autonomous driving mode. For instance, a notification from a vehicle of the fleet identifying a feature and a location of the feature may be received. A first bound for a scouting area may be identified based on the location of the feature. A second bound for the scouting area may be identified based on a lane closest to the feature. A scouting objective may be generated for the feature based on the first bound and the second bound.

Abstract: Aspects of the disclosure relate to detecting and responding to malfunctioning traffic signals for a vehicle having an autonomous driving mode. For instance, information identifying a detected state of a traffic signal for an intersection. An anomaly for the traffic signal may be detected based on the detected state and prestored information about expected states of the traffic signal. The vehicle may be controlled in the autonomous driving mode based on the detected anomaly.