Abstract: A method includes dividing an image into a plurality of patches; generating for the patches image tokens each including a halting score associated with the image token; generating a contextual features token representative of at least one contextual feature in connection with the image; processing the image tokens and the contextual features token using a vision transformer block; performing adaptive halting on the image tokens output from the transformer block, the adaptive halting comprising updating the halting scores associated with the image tokens and discarding ones of the image tokens having halting scores greater than or equal to a predetermined threshold score, wherein the non-discarded image tokens comprise remaining image tokens; and forwarding the remaining image tokens to a next processing block.

Abstract: A computer vision module facilitates multi-sensor vision of AVs based on privileged information learning. The module may input first sensor data captured by a first sensor into an already-trained privileged model, input second sensor data captured by a second sensor into a first backbone of a target model, and input third sensor data captured by a third sensor into a second backbone of the target model. The three sensors may be of different types. To train the target model, internal parameters of the target model may be modified to minimize a loss, which may include a privilege loss, which indicates a difference between a latent representation of the first sensor data and a latent representation of the second sensor data, and another privileged loss, which indicates a difference between the latent representation of the first sensor data and a latent representation of the third sensor data.

Abstract: An operation of an AV in a scene is simulated. The AV includes sensors detecting the scene and generating sensor data. The sensor data can be input into a control model that outputs control signals, in accordance with which the AV operates. A learning cost, i.e., a cost of training or applying the control model, is determined. A performance of the AV during the simulation is evaluated. A simulation cost, i.e., a cost of running the simulation is determined. The control model can be optimized based on the learning cost and the performance of the AV. Settings of the sensors can be optimized based on the simulation cost and the performance of the AV. The optimization of the control model or the settings of the sensors can be done through reinforcement learning.

Abstract: Apparatus and methods of the present disclosure provide a solution to quickly sort through and identify relevant data sets from large amounts of data collected by numerous vehicles. This process is a form of data mining where a processor executes instructions out of a memory to evaluate sensor data collected by vehicles over time. These vehicles may be driven by person's such that data collected by those sensors would be representative of how a person drives. After data has been acquired, it may be evaluated to identify portions of data where one vehicle encounters another vehicle along a roadway. Data may be classified as being associated with particular types of driving maneuvers and organized into data sets with labels consistent with the classification or driving maneuver. Classifications could correspond to different typed of driving maneuvers, “turning left across traffic,” “all way stop,” “merging onto a highway,” or “lane change”.

Abstract: Color filter array patterns are provided for enhancing an image sensor's light sensitivity while preserving a color accuracy for image signal processing applications. In one example, an image sensor can include a substrate layer containing a first set of photodiodes and a second set of photodiodes, wherein each of the first set of photodiodes is larger than each of the second set of photodiodes; a first color filter array (CFA) covering the first set of photodiodes, wherein the first CFA includes a first set of color filters and a portion of the first set of color filters includes one or more clear filters; a second CFA covering the second set of photodiodes, wherein the second CFA includes a second set of color filters that is different than the first set of color filters; and one or more lenses covering the first CFA and the second CFA.

Abstract: Systems and methods for a vehicle to use a fallback control system orthogonal to a primary control system are provided. A method includes collecting, by a fallback control system of a vehicle, sensor data generated by one or more sensors of the vehicle; determining, by the fallback control system using a first process different from a second process of a primary control system of the vehicle, an alternative action for the vehicle based on the sensor data; and sending, by the fallback control system based on the alternative action, an instruction for modifying an action associated with the primary control system upon a faulty condition associated with the primary control system.

Abstract: An autonomous vehicle (AV) is described herein. The AV is configured to identify an occluded region where a portion of a field of view of a sensor is occluded by an object. The AV is further configured to hypothesize that an object exists in the occluded region and is moving in the occluded region. The AV is still further configured to perform a driving maneuver based upon the hypothesized object existing in the occluded region.

Abstract: Systems and methods for collision imminent detection are provided. A method includes collecting sensor data generated by one or more sensors of a vehicle; detecting, by a fallback control system of the vehicle based on the sensor data, an imminent collision between the vehicle and an object in an environment of the vehicle using a first process different from a second process of a primary control system of the vehicle, wherein the first process and the second process are associated with at least one of a perception or a prediction; and transmitting, by the fallback control system to the primary control system, an indication of the detected imminent collision.

Abstract: System, methods, and computer-readable media for a training technique of an object trajectory prediction model that inputs semantic map data pertaining to an environment and trajectory data of an object into an autoencoder neural network, which outputs the original trajectory as a feature embedding vector. Each feature embedding vector serves as a unique identifier for each behavior. Cluster analysis is performed on feature embedding vectors to determine clusters, each associated with a particular behavior attribute.

Abstract: Particular embodiments described herein provide for a system and method for determining an intent of a third-party autonomous vehicle by an autonomous vehicle. The method can include identifying the third-party autonomous vehicle, identifying a third-party autonomous vehicle behavior signature associated with the third-party autonomous vehicle, capturing sensor data related to the third-party autonomous vehicle, analyzing the captured sensor data to determine behaviors of the third-party autonomous vehicle, and using a third-party autonomous vehicle behavior signature to determine if the behaviors of the third-party autonomous vehicle indicate a specific intent of the third-party autonomous vehicle. The third-party autonomous vehicle behavior signature can be created by identifying behaviors of the third-party autonomous vehicle and associating the behaviors with an intent of the third-party autonomous vehicle.

Abstract: The disclosed technology provides methods for training of a convolutional neural network (CNN) to identify or predict its own errors, and then those errors are used as inputs with feature visualization to generate images of scenes associated with those errors. This allows adjustment of a set of labeled training images, and then the adjusted set of labeled training images are used to retrain or further train the CNN. Systems and machine-readable media are also provided.

Abstract: Disclosed are embodiments for facilitating a backup system for observability, communication, and control of autonomous systems. In some aspects, an embodiment includes initializing an instance of a backup tool to provide observability, communication, and control of a fleet of autonomous vehicles (AVs) responsive to an outage of primary fleet tool services; storing AV data received from the fleet of AVs in in-memory storage of the instance of the backup tool; populating a cache of the backup tool with semi-static data corresponding to the fleet of AVs; authenticating a user requesting access to the instance, the authenticating to utilize credentials of the user established with the primary fleet tool services; and facilitating the issuance of one or more control commands to the fleet of AVs based on the AV data stored in the in-memory storage and based on the semi-static data stored in the cache.

Abstract: A radar apparatus at a vehicle may transmit a set of radar energy that is reflected off an object after which the reflected radar energy may be received at the radar apparatus. When relative motion of a radar apparatus and an object include movement in two different directions, power of the reflected radar signals may be spread out in a manner that makes data associated with the reflected radar signals difficult to interpret. This spreading out of the radar signals can make mappings of the received radar data appear to be smeared or distorted. To compensate for this smearing effect, mappings of this smeared radar data may be compared with curves from which compensation factors may be identified. These compensation factors may allow a processor to perform calculations to generate updated mappings of the radar data that may allow the processor to more accurately identify characteristics of the object.

Abstract: Systems and techniques are described for Light Detection and Ranging (LiDAR) noise modeling using machine learning (ML). An example method can include collecting, using one or more sensors, a first set of data for a simulation environment and generating, using the first set of data, a point cloud that represents and/or describes the simulation environment. The method can further include collecting, using the one or more sensors, a second set of data for a real-world environment and generating a noise model using the second set of data and a neural network. The method can also include generating, using the noise model and the point cloud, a noisy point cloud that represents and/or describes the real-world environment.

Abstract: The subject disclosure relates to features that improve safety for autonomous vehicle (AV) driving maneuvers. In some aspects, a process of the disclosed technology includes steps for detecting an unprotected maneuver, navigating an AV into an intersection, and detecting a wheel safety angle relative to the intersection. In some aspects, the process further includes steps for automatically adjusting a wheel angle of the AV based on the wheel safety angle. Systems and machine-readable media are also provided.

Abstract: The present technology pertains to validating a virtual camera in a simulation environment by utilizing an improved camera model to provide quantitative measurements of the model's performance. A method of validating a virtual camera in a simulation environment comprises presenting at least one reference chart in the simulated environment and capturing images of the reference chart in the simulated environment using a virtual camera. The method further includes interrupting an image pipeline of the virtual camera after at least one simulated process in the image pipeline to extract a RAW image. The method analyzes the RAW image to derive measurements of metrics to characterize the virtual camera. The measured metrics are compared to metrics of a calibrated real-world camera to verify that the metrics are within a threshold delta that is indicative that the virtual camera sufficiently approximates the real-world camera.

Abstract: System, methods, and computer-readable media for configuring an autonomous vehicle based on safety scores determined by a safety score prediction algorithm, and an associated training technique, to output a safety score for a predicted and/or actual path. The safety score is a value indicating a likelihood of risky events per distance. The safety score prediction algorithm is trained with historical human driving datasets associated with paths (predicted or actual) taken by one or more AVs during human driving.

Abstract: Systems and techniques are provided for communicating using an automotive audio bus. An example method can include receiving, by a first automotive audio bus manager, a request from a first automotive audio bus client for a first data transmission to a second automotive audio bus client, wherein a first size of the first data transmission exceeds a maximum data frame size; parsing the first data transmission into a first plurality of data frames based on the first size of the first data transmission and the maximum data frame size; sending a first header frame to a second automotive audio bus manager associated with the second automotive audio bus client, wherein the first header frame includes a first indication of a first incoming long message transmission and the first size of the first data transmission; and sending the first plurality of data frames to the second automotive audio bus manager.

Abstract: Approaches for top speed dependent operational parameter configuration are disclosed. An operational parameter to be used by an autonomous vehicle is received. Operational modes and corresponding operational regions are determined based on the received operational parameter. A first set of autonomous vehicle parameters are determined and set based on the received operational parameter. A second set of autonomous vehicle parameters are determined and set based on the first set of autonomous vehicle parameters.

Abstract: A method includes using sensory data produced by at least one onboard sensor of a witness vehicle to predict an imminent collision in an environment of the witness vehicle, the predicted imminent collision involving at least one participant vehicle; and alerting the least one participant vehicle to the predicted imminent collision, wherein the alerting is performed by the witness vehicle and prompts the at least one participant vehicle to perform at least one action to mitigate the predicted imminent collision.