Abstract: A computer system for rendering a graphical user interface for visualising runs of a driving scenario in which an ego agent navigates a road layout, comprising an input configured to receive a map of the road layout and run data comprising a sequence of timestamped ego agent states and a time-varying numerical score quantifying the performance of the ego agent with respect a set of run evaluation rules; and a rendering component configured to cause a graphical user interface to display, for each rule: a plot of the time-varying numerical score, and a marker denoting a selected time index of the plot, the marker movable along the time axis to change the selected time index; and a scenario visualization comprising a visualization of the run at the selected time index, whereby moving the marker along the time axis causes the scenario visualisation to update as the time index is changed.

Abstract: To locate and model a moving 3D object captured in a time sequence of multiple images, semantic keypoint detection is applied to each image, in order to generate a set of 2D semantic keypoint detections in an image plane of the image. For each image, a camera pose and an initial object pose is received. Based on the initial 3D model, and the initial object pose and the camera pose for each image, a set of keypoint projections is computed in the image plane of the image, the keypoint projections being 2D projections of 3D semantic keypoints of the initial 3D model. A new 3D model and a new object pose are determined for each image based on an aggregate reprojection error between the 2D semantic keypoint detections and the keypoint projections across the time sequence of multiple images.

Abstract: A computer system for testing a real-time perception system for a sensor-equipped vehicle, the computer system comprising: an input configured to receive data of a real-world driving run performed by a sensor-equipped vehicle comprising a time series of sensor data and at least one associated time series of run-time perception outputs extracted therefrom by the perception system; a rendering component configured to render a GUI comprising a perception error timeline having a visual indication of any perception error that occurred at each time step of the run; a ground truthing pipeline configured to process at least one of the sensor data and the run-time perception outputs, by applying a non-real-time and/or non-causal perception algorithm thereto, to extract ground truth perception outputs; and a perception oracle configured to compare the run-time perception outputs with the ground-truth perception outputs, and identify any perception errors that occurred for generating the perception error timeline.

Abstract: A method of testing a candidate perception setup for an autonomous vehicle, comprising: receiving ground truth of a driving scenario run; providing a time-sequence of ground truth snapshots of the driving scenario run, wherein a decision making component independent of the ego agent determines a first time-sequence of decisions for the ground truth snapshots; providing a time-sequence of ablated snapshots of the driving scenario run, each ablated snapshot generated so as to cause a perception error that is representative of the candidate perception setup, wherein the decision making component determines a second time-sequence of decisions for the ablated snapshots; and computing a similarity measure between the first and second time-sequences of decisions denoting an extent to which the candidate perception setup caused a change in one or more decisions points of the second time-sequence relative to the first time-sequence, a decision point occurring when a decision changes between adjacent timesteps.

Abstract: A computer implemented method of generating a scenario to be run in a simulation environment for testing the behaviour of an autonomous vehicle includes rendering on the display of a computer device an interactive visualisation of a scenario model for editing. The scenario model includes one or more interactions between an ego vehicle object and one or more dynamic challenger objects, each interaction defined as a set of temporal and/or relational constraints between the dynamic ego object and at least one of the challenger objects. The scenario model comprises a scene topology and the interactive visualisation comprises scene objects including the ego vehicle and the at least one challenger object displayed in the scene topology. The scenario is associated with a timeline extending in a driving direction of the ego vehicle relative to the scene topology.

Abstract: An encoder is trained together with a perception component based on a training set comprising unannotated sensor data sets and annotated sensor data sets in a sequence of multiple training steps.

Abstract: A computer-implemented method of determining control signals for controlling an autonomous vehicle to implement a slowdown manoeuvre, comprising: detecting an obstacle at a distance ahead of the autonomous vehicle; comparing the distance with a threshold value and implementing a slowdown manoeuvre in dependence on the comparison, the slowdown manoeuvre selected from: a first slowdown manoeuvre carried out by a kinematic function, which is a time derivative of acceleration, in which a constraint optimisation has been applied to optimise a cost function of the slowdown manoeuvre subject to a set of hard constraints that require a final acceleration, speed and position to satisfy respective acceleration, speed and position targets, given an initial speed and acceleration of the vehicle, and impose a jerk magnitude upper limit; and a second slowdown manoeuvre implemented in an adaptive cruise control mode which aims to reach a target headway between the autonomous vehicle and the obstacle.

Abstract: A computer implemented method of evaluating planner performance for an ego robot, the method comprising: receiving first run data of a first run, the run data generated by applying a planner in a scenario of that run to generate an ego trajectory taken by the ego robot in the scenario; extracting scenario data from the first run data to generate scenario data defining the scenario; providing the scenario data to a simulator configured to execute a simulation using the scenario data and implementing a second planner to generate second run data; comparing the first run data and the second run data to determine a difference in at least one performance parameter; and generating a performance indicator associated with the run, the performance indicator indicating a level of the determined difference between the first run data and the second run data.

Abstract: A computer-implemented method of planning ego actions for a mobile robot in the presence of at least one agent, comprising: searching for an optimal ego action in multiple search steps, each comprising: selecting an ego action from a set of possible ego actions, selecting an agent behaviour from a set of possible agent behaviours, running a simulation based on the selected ego action and agent behaviour, determining a possible outcome, and assigning a reward to the selected ego action, based on a reward metric, wherein selection of the ego action in later search steps is biased towards higher reward ego action(s) but selection of the agent behaviour in later search steps is biased towards riskier agent behaviour(s), a risky agent behaviour being, according to earlier search steps, more likely to result in a lower reward outcome and choosing an ego action based on the rewards computed in the search steps.

Abstract: A computer-implemented method of evaluating the performance of a trajectory planner for a mobile robot in a real or simulated scenario, the method comprising: receiving scenario ground truth of the scenario, the scenario ground truth generated using the trajectory planner to control an ego agent of the scenario responsive to at least one other agent of the scenario, and comprising an ego trace of the ego agent and an agent trace of the other agent; evaluating the ego trace, by a test oracle, in order to assign at least one time series of test results to the ego agent, the time-series of test results pertaining to at least one performance evaluation rule; extracting one or more predetermined blame assessment parameters based on the agent trace; and applying one or more predetermined blame assessment rules to the blame assessment parameters, and thereby determining whether failure on the at least one performance evaluation rule is acceptable.

Abstract: A computer-implemented method of filtering noisy observations of a system to estimate a state of the system, the method comprising: applying a filter to a sequence of observations based on one or more filter parameters, to compute a set of system states, the filter parameters configurable to change respective contributions of the observations to the set of system states; wherein the filter is applied in multiple iterations with different values of the configurable parameter(s), to update the set of system states, wherein the different values of the configurable filter parameter are determined via a gradient-based optimization of a loss function that penalizes values of the configurable parameters that result in relatively large contributions from outlier observations that deviate from the set of system states by a relatively large amount.

Abstract: A computer system for testing the performance of a stack for planning ego vehicle trajectories in real or simulated driving scenarios, the computer system comprising: at least a first input configured to receive (i) scenario ground truth and (ii) internal state data of the stack, the scenario ground truth and internal state data generated using the stack to control an ego agent responsive to at least one other agent in the simulated driving scenario; at least a second input configured to receive a defined operational design domain (ODD); a test oracle configured to apply one or more driving rules to the scenario ground truth for evaluating the performance of the stack in the scenario, and provide an output for each of the driving rules indicating whether that driving rule has been complied with; wherein the one or more driving rules include at least one ODD-based response rule, the test oracle configured to apply the ODD-based response rule by: processing the scenario ground truth over multiple time steps, to

Abstract: A computer-implemented method of evaluating the performance of a trajectory planner in simulation comprises miming first instances of a scenario in a simulator, the first instances run with a first set of parameterizations of the scenario, the trajectory planner used to control an ego agent responsive in each scenario instance; evaluating performance of the trajectory planner in each scenario instance, thereby computing a set of test results for the first set of scenario parameterizations; identifying at least one target parameterization of the first set based on the set of test results; and based on the target parameterization, determining a second set of parameterizations of the scenario for miming second instances of the scenario for exploring a subspace of the parameter space in the vicinity of the target parameterization.

Abstract: A computer implemented method for generating a simulation environment for testing an autonomous vehicle, the method comprising generating a scenario comprising a dynamic interaction between an ego object and challenger object, the interaction being defined relative to a static scene topology. The method comprises providing a dynamic layer comprising parameters of the dynamic interaction and a static layer comprising the static scene topology to a simulator, searching a store of maps to access a map having a matching scene topology to the static scene topology, and generating a simulated version of the dynamic interaction using the matching scene topology.

Abstract: It A computer implemented method of training an encoder to extract features from sensor data comprises training a machine learning (ML) system based on a self-supervised loss function applied to a training set, the ML system comprising the encoder. The training set comprises first data representations and corresponding second data representations, wherein the encoder extracts features from each first and second data representation, and wherein the self-supervised loss function encourages the ML system to associate each first data representation with its corresponding second data representation based on their respective features.

Abstract: A computer implemented method of training an encoder to extract features from sensor data comprises training a machine learning (ML) system based on a self-supervised loss function applied to a training set, the ML system comprising the encoder. The training set comprises sets of real sensor data and corresponding sets of synthetic sensor data. The encoder extracts features from each set of real and synthetic sensor data, and the self-supervised loss function encourages the ML system to associate each set of real sensor data with its corresponding set of synthetic sensor data based on their respective features.

Abstract: An in-stream disparity processing system comprises a delay block having an input for receiving an input stream of disparity cost vectors, and configured to provide a delayed stream of disparity cost vectors at an output of the delay block, by delaying the input stream by a predetermined amount; and a processing block having a cost input connected to receive the delayed stream of disparity cost vectors and a smoothing input connected to receive the input stream of disparity cost vectors, and configured to apply cost smoothing to the delayed stream based on the input stream, so as to generate, at an output of the processing block, a stream of reverse pass disparity cost vectors.

Abstract: An occlusion metric is computed for a target object in a 3D multi-object simulation. The target object is represented in 3D space by a collision surface and a 3D bounding box. In a reference surface defined in 3D space, a bounding box projection is determined for the target object with respect to an ego location. The bounding box projection is used to determine a set of reference points in 3D space. For each reference point of the set of reference points, a corresponding ray is cast based on the ego location, and it is determined whether the ray is an object ray that intersects the collision surface of the target object. For each such object ray, it is determined whether the object ray is occluded. The occlusion metric conveys an extent to which the object rays are occluded.

Abstract: A method of annotating known objects in road images captured from a sensor-equipped vehicle, the method implemented in an annotation system and comprising: receiving at the annotation system a road image containing a view of a known object; receiving ego localization data, as computed in a map frame of reference, via localization applied to sensor data captured by the sensor-equipped vehicle, the ego localization data indicating an image capture pose of the road image in the map frame of reference; determining, from a predetermined road map, an object location of the known object in the map frame of reference, the predetermined road map representing a road layout the map frame of reference, wherein the known object is one of: a piece of road structure, and an object on or adjacent a road; computing, in an image plane defined by the image capture pose, an object projection, by projecting an object model of the known object from the object location into the image plane; and storing, in an image database, image

Abstract: A computer-implemented method of modelling a perception system, the perception system configured to receive sensor data and interpret the sensor data to generate actual perception outputs, comprises: receiving a plurality of input samples, wherein each input sample comprises sensor data and is associated with one or more training perception ground truths pertaining to one or more ground truth objects; providing the sensor data of each input sample to the perception system to be modelled, wherein the perception system interprets the sensor data, in order to generate one or more actual perception outputs for the input sample; and training a function approximator to model the perception system by: for each input sample, inputting the training perception ground truths to the function approximator, wherein the function approximator computes one or more predicted perception values by processing the training perception ground truths but not the sensor data from which the actual perception outputs are generated, and