Abstract: Sensors coupled to a vehicle are calibrated, optionally using a dynamic scene with sensor targets around a motorized turntable that rotates the vehicle to different orientations. One vehicle sensor captures a representation of one feature of a sensor target, while another vehicle sensor captures a representation of a different feature of the sensor target, the two features of the sensor target having known relative positioning on the target. The vehicle generates a transformation that maps the captured representations of the two features to positions around the vehicle based on the known relative positioning of the two features on the target.

Abstract: System, methods, and computer-readable media for an object path prediction model, and an associated training technique, to output a path that is considered an object collision path. The object path prediction model outputs a set of predicted paths for the object that are outputted to a trained planning algorithm, which includes paths that are most likely to occur and a path that is considered an object collision path. The predicted paths are sent to the trained planning algorithm and used for planning a trajectory for the autonomous vehicle that is associated with a low probability of colliding with or taking a sudden evasive action to avoid the object.

Abstract: Systems and techniques are provided for management of autonomous cargo by autonomous vehicles (AVs). An example method can include determining, based on data from one or more sensors, a location for deploying a ramp that enables robots to enter the AV, the location comprising an area free of obstacles having one or more dimensions above a threshold; generating an instruction configured to trigger the AV to stop at the location; based on a determination that the AV is at the stopping position, deploying the ramp; sending, to the robots, a message instructing the robots to enter a cabin of the AV via the ramp and guiding each robot to a respective location within the cabin; and based on a determination that the AV has reached a destination of one or more robots, deploying the ramp and guiding the one or more robots to exit the cabin via the ramp.

Abstract: For some embodiments of the present disclosure, systems and methods for using computer vision to guide processing of receive responses of RADAR sensors of a vehicle are described. A computer implemented method comprises receiving, with one or more receivers of RADAR sensors, environment receive RF responses, receiving region of interest (ROI) information including segments of a tentative set of ROIs with free space area boundaries from an image processor of an image processing chain for the computer vision, dynamically determining a number of polyphase filters based on a number of ROIs having dynamic scenes that are provided by the image processing chain, and adjusting parameters of the polyphase filters of a RADAR processing chain based on the ROI information.

Abstract: An AV may detect degraded states of other AVs. For instance, the AV may use its sensors to detect one or more other vehicles in the local area. The AV may determine whether any of these vehicles is a peer of the AV based on features of these vehicles captured by its sensors. Alternatively, the AV may determine if any vehicle is a peer based on encrypted communication with the vehicle. The AV may also determine whether any peer AV has a state that deviates from an expected or desirable state. The AV may communicate with an online system and request the online system to provide information for identifying the vehicle or detecting degraded states. The AV may also report degradations in peer AVs to the online system. The online system may recover the degraded AV or dispatch another AV to complete a task assigned to the degraded AV.

Abstract: A unified electronic control unit (ECU) interface provides a bridge from ECU clients to the ECU. The ECU interface may be implemented on a vehicle's onboard computer, and it may be used by clients executing on the onboard computer and, in some cases, by clients outside the onboard computer. The ECU interface may receive a request from a client in a first messaging format, e.g., a local networking protocol used by processes running on the vehicle. The ECU interface translates the request into a second messaging format and transmits the request to the ECU. The ECU interface receives a response from the ECU and translates the response into the first messaging format for delivery to the client. The ECU interface may further perform authentication of the client and/or the ECU, packet serialization of the response from the ECU, packet diagnostics, and other functionalities associated with communicating with the ECU.

Abstract: A set of devices are coupled to a control area network (CAN) bus. At least some of the devices include switchable resistors that can apply a termination resistor to the CAN bus. Identification pins are used to identify devices on the CAN bus, and also to identify which device or devices are at the ends of the CAN bus. If a device is at an end of the CAN bus, the switch of the switchable resistor is set to couple the termination resistor to the CAN bus. Otherwise, the termination resistor is not coupled to the CAN bus.

Abstract: Disclosed are embodiments for facilitating purposeful stress testing of autonomous vehicle response time with simulation. In some aspects, an embodiment includes receiving a request to launch a simulation scenario on an autonomous vehicle (AV) that is to operate on a real-world test course; initiating a simulation derived from the simulation scenario using a simulation driver that is executing on the AV; engaging operation of the AV on the real-world test course; coordinating a simulated AV position in the simulation with a physical AV position of the AV on the real-world test course; combining virtual entities from the simulation with physical entities on the real-world test course into a list of tracked objects for the AV; and causing the AV to respond to the list of tracked objects including the virtual entities and the physical entities during the operation of the AV on the real-world test course.

Abstract: An ultrasonic detection system provides enhanced object detection capabilities. The ultrasonic detection system can generate complex ultrasonic waveforms based on excitation voltage waveforms constructed using high frequency components. In certain embodiments, an ultrasonic detection system may be capable of producing multiple different waveforms, e.g., different waveforms output by different transmitters, or multiple time-multiplexed waveforms output by the same transmitter. The ultrasonic detection system may be included in an autonomous vehicle (AV) for object detection in the environment of the AV.

Abstract: Autonomous vehicles utilize perception to identify objects in a vehicle's surrounding environment and plan a trajectory. In perceiving an AV's surroundings, an object detection model uses sensor data from AV sensors to detect objects in the AV surroundings. An object detection model can be trained to detect objects using training datasets having sensor data and ground truth objects. In training, the object detection model uses the sensor data to identify objects in the locations of the ground truth objects. However, sometimes there is little or no sensor data corresponding to the ground truth objects, and the model is not able to learn patterns in sensor data to correctly detect the objects. Systems and methods are provided herein for removing ground truth objects that do not have sufficient corresponding sensor data from the training dataset, thereby preventing these objects from dominating the loss and interfering with the model learning useful patterns.

Abstract: Autonomous vehicles (AVs) utilize perception and understanding of objects on the road to predict behaviors of the objects, and to plan a trajectory for the vehicle. In some situations, an object may be occluded and undetected by an AV. However, a different AV viewing the same scene may detect the object. With information from multiple views of the same scene, it is possible to determine occlusion attributes for the object, such as relational occlusion information and extent of occlusion. For the AV that is driving on the road, having knowledge of the occluded object and the occlusion attributes can improve the performance of perception, understanding, tracking, prediction, and/or planning algorithms. For the algorithms, occlusion attributes can be generated from the multi-view data and included as part of labeled data for machine learning training. The models in the algorithms can learn to better handle occluded objects.

Abstract: Systems and methods for automatic removal for selected training data from a dataset are provided. Systems and methods are provided for identifying portions of radar data that contain information that is not present in camera and/or lidar data. The identified portions of the radar data can be processed by a neural network to provide the additional information. The identified parts of the radar data can then be processed while the remaining parts of the radar data remain unprocessed. In various examples, features can be extracted from identified regions of the radar data using a neural network and fused with camera and/or lidar features. In some examples, the fused features can be used for object detection.

Abstract: Autonomous vehicles (AVs) utilize perception and understanding of objects on the road to predict behaviors of the objects, and to plan a trajectory for the vehicle. In some scenarios, an AV may benefit from having information about an object that is not within a perceivable area of the sensors of the AV. Another AV in the area that has the object within a perceivable area of the sensors of the other AV may share information with the AV. Rich information about the object determined by the other AV can be compressed using an encoder and transferred efficiently to the AV for inclusion in a temporal map that combines locally determined object information and transferred object information determined by other AV(s). The temporal map may be used in one or more parts of the software stack of the AV.

Abstract: A computer vision module facilitates multi-sensor vision of AVs based on privileged information learning. The module may input first sensor data captured by a first sensor into an already-trained privileged model, input second sensor data captured by a second sensor into a first backbone of a target model, and input third sensor data captured by a third sensor into a second backbone of the target model. The three sensors may be of different types. To train the target model, internal parameters of the target model may be modified to minimize a loss, which may include a privilege loss, which indicates a difference between a latent representation of the first sensor data and a latent representation of the second sensor data, and another privileged loss, which indicates a difference between the latent representation of the first sensor data and a latent representation of the third sensor data.

Abstract: A method includes dividing an image into a plurality of patches; generating for the patches image tokens each including a halting score associated with the image token; generating a contextual features token representative of at least one contextual feature in connection with the image; processing the image tokens and the contextual features token using a vision transformer block; performing adaptive halting on the image tokens output from the transformer block, the adaptive halting comprising updating the halting scores associated with the image tokens and discarding ones of the image tokens having halting scores greater than or equal to a predetermined threshold score, wherein the non-discarded image tokens comprise remaining image tokens; and forwarding the remaining image tokens to a next processing block.

Abstract: An operation of an AV in a scene is simulated. The AV includes sensors detecting the scene and generating sensor data. The sensor data can be input into a control model that outputs control signals, in accordance with which the AV operates. A learning cost, i.e., a cost of training or applying the control model, is determined. A performance of the AV during the simulation is evaluated. A simulation cost, i.e., a cost of running the simulation is determined. The control model can be optimized based on the learning cost and the performance of the AV. Settings of the sensors can be optimized based on the simulation cost and the performance of the AV. The optimization of the control model or the settings of the sensors can be done through reinforcement learning.

Abstract: System, methods, and computer-readable media for a training technique of an object trajectory prediction model that inputs semantic map data pertaining to an environment and trajectory data of an object into an autoencoder neural network, which outputs the original trajectory as a feature embedding vector. Each feature embedding vector serves as a unique identifier for each behavior. Cluster analysis is performed on feature embedding vectors to determine clusters, each associated with a particular behavior attribute.

Abstract: Color filter array patterns are provided for enhancing an image sensor's light sensitivity while preserving a color accuracy for image signal processing applications. In one example, an image sensor can include a substrate layer containing a first set of photodiodes and a second set of photodiodes, wherein each of the first set of photodiodes is larger than each of the second set of photodiodes; a first color filter array (CFA) covering the first set of photodiodes, wherein the first CFA includes a first set of color filters and a portion of the first set of color filters includes one or more clear filters; a second CFA covering the second set of photodiodes, wherein the second CFA includes a second set of color filters that is different than the first set of color filters; and one or more lenses covering the first CFA and the second CFA.

Abstract: An autonomous vehicle (AV) is described herein. The AV is configured to identify an occluded region where a portion of a field of view of a sensor is occluded by an object. The AV is further configured to hypothesize that an object exists in the occluded region and is moving in the occluded region. The AV is still further configured to perform a driving maneuver based upon the hypothesized object existing in the occluded region.

Abstract: Systems and methods for a vehicle to use a fallback control system orthogonal to a primary control system are provided. A method includes collecting, by a fallback control system of a vehicle, sensor data generated by one or more sensors of the vehicle; determining, by the fallback control system using a first process different from a second process of a primary control system of the vehicle, an alternative action for the vehicle based on the sensor data; and sending, by the fallback control system based on the alternative action, an instruction for modifying an action associated with the primary control system upon a faulty condition associated with the primary control system.