Abstract: A vehicle operating system (VOS) in an autonomous driving vehicle (ADV) can communicate with a cloud platform to automatically train AI models. The VOS collects real-time data from the ADV, and generates inference data based on the real-time data using a teacher edge model of an AI model and generates second inference data based on the real-time data using a student edge model of the AI model. The VOS then obtains one or more differences between the first inference data and the second inference data, and retrains the student edge model of the AI model based on the one or more differences. Both real-time data and the retrained student edge model are uploaded to a cloud platform for use in upgrading the student edge model and the teacher edge model on the cloud platform. The upgraded teacher edge model and the student edge model can be redeployed over-the-air (OTA) through a software define process. The above process of training AI models can be repeated in a closed-loop automatically without user intervention.

Abstract: A video system includes a first data gathering node configured to receive a plurality of image streams from a plurality of cameras, respectively. Each of the plurality of cameras captures an environment of an autonomous driving vehicle (ADV). The first data gathering node tags the plurality of image streams with metadata that identifies each of the plurality of image streams, and combines the plurality of image streams with the metadata to form a combined image stream. A second data gathering node is communicatively coupled to the first data gathering node and is to receive the combined image stream from the first data gathering node and output the combined image stream with a second combined image stream.

Abstract: A hardware unit of an ADV comprises an input port to directly receive data from one or more sensors perceiving a driving environment. The hardware unit is coupled with the one or more sensors to perform data processing of the data from one or more sensors. The hardware unit comprises a monitor unit to monitor a data rate of output data after the data processing. The hardware unit further comprises a self-adjustment unit to dynamically configure and adjust the data processing based on the data rate of output data. The hardware unit further comprises an output port to transfer the output data to an autonomous driving system (ADS) of the ADV to control the ADV to drive autonomously based on the output data.

Abstract: An autonomous driving vehicle (ADV) includes a computer network that includes at least one wireless gateway. The ADV also includes a plurality of network components coupled to the computer network, the plurality of network components including a target device. The ADV also includes a processor, configured to receive, through the at least one wireless gateway, a firmware to be installed on the target device, and to direct the firmware to the target device through a first network path of the computer network. In response to detecting a failure to update the target device with the firmware, the processor directs the firmware to the target device through a second network path of the computer network.

Abstract: In one aspect, a computing device of an autonomous driving vehicle (ADV) is configured to determine a first control signal for a light detection and ranging (Lidar) sensor of the ADV and a second control signal for a camera of the ADV, provide the first control signal to the Lidar sensor and the second control signal to the camera, and process Lidar output of the Lidar sensor and camera output of the camera to detect one or more features of the Lidar output or camera output. In response to detecting the one or more features, the computing device is to adjust the first control signal or the second control signal.

Abstract: A sensor system for an autonomous driving vehicle (ADV) includes a sensor interface coupled to a plurality of sensors, a host interface coupled to a host system of the ADV, and a self-adaptive sensor transfer unit coupled between the sensor interface and the host interface. The self-adaptive sensor transfer unit includes a sensor monitor module, configured to monitor a data rate of sensor data received from a sensor, and a configuration control module, configured to: receive a target data rate from the host via the host interface; receive the monitored data rate of the sensor data; and control the data rate of the sensor data to be within a threshold of the target data rate.

Abstract: A cost-latency balanced method of processing camera image data in an autonomous driving vehicle (ADV) is described. The ADV includes a main compute unit coupled to an FPGA unit and a graphical processing unit (GPU). The method includes receiving, by the main compute unit, a full raw image data and a partial compressed image data from the FPGA unit, the full raw image being raw image data captured by all cameras mounted on the ADV, and the partial compressed image data being compressed from a partial raw image data captured by a subset of the cameras mounted on the ADV. The method further includes transmitting the partial compressed image data to a remote driving operation center; and consuming the full raw image data for environment perception, and the full raw image data is also compressed into a full compressed image data by the GPU for use in offline processing.

Abstract: A system obtains a performance profile corresponding to times taken to perform an inferencing by a machine learning (ML) model using a different number of processing resources from a plurality of processing resources. The system determines one or more groupings of processing resources from the plurality of processing resources, each grouping includes one or more partitions. The system calculates performance speeds corresponding to each grouping based on the performance profile. The system determines a grouping having a best performance speed from the calculated performance speeds. The system partitions the processing resources based on the determined grouping to perform the inferencing.

Abstract: A system determines a timing matrix corresponding to inference times taken for a number of machine learning (ML) models to be executed by a number of processing resources of a computing device. The processing resources includes at least a first and a second type of processing resources. The system applies a service-specific model-first scheduling scheme or a service-specific hardware-first scheduling scheme to obtain corresponding service-specific mappings. The system determines a best mapping from the corresponding service-specific mappings. The system schedules each of the ML models to a corresponding processing resource from the processing resources according to the best mapping. The system executes the ML models using corresponding mapped processing resources.

Abstract: The present disclosure provides a system and method that retrieves a plurality of logic blocks and a plurality of safety levels corresponding to the plurality of logic blocks. The system and method determines, by a processor, which of the plurality of logic blocks require one or more redundant logic blocks based on their corresponding safety level. The system and method produces a logic design based on the plurality of logic blocks and the one or more redundant logic blocks.

Abstract: A multiprocessor unit (MPU) in an autonomous driving vehicle (ADV) can provide hard real-time performance. In an embodiment, the MPU can include a hypervisor used to virtualize multiple cores of the MPU, which can further be partitioned into two sets of cores that are isolated from each other. The first set of cores are designated to run real-time related services as trusted applications directly on the hypervisor, and the real-time related services are given higher priority than kernel-level threads on the first set of cores. The second set of cores are designated to run a kernel of an operating system (e.g., Linux). Further, the kernel is patched using a hard real-time open source package to achieve hard real-time performance. An open source package can be used for interprocess communication (IPC) between different electronic control units (ECU) in the ADV.

Abstract: In one embodiment, a vehicle operating system (VOS) that can be partially ported to different types of microcontroller units (MCUs) includes at least one multiprocessor unit (MPU) with an operating system kernel running thereon, and at least one microcontroller unit (MCU) with multiple cores. Each core includes a set of unified application programming interfaces (APIs) for loading one or more MCU drivers corresponding to a type of the MCU, and one or more I/O drivers corresponding to a type of each of the one or more I/O devices associated with the MCU. The set of unified APIs includes at least one API for each service, and can vertically integrate a device path for the service from a hardware layer of the core to the service layer of the core. The VOS further includes multiple pairs of hardware-protected memories associated with each core to enable interprocess communication between the cores.

Abstract: In an embodiment, a method verifies functionality of computation hardware on a device under test (DUT). The method loads a software test program onto the DUT, wherein the DUT includes computation hardware components. The method executes the software test program on the DUT to test the hardware components. During the testing, the software test program instructs one or more devices external to the DUT to provide one or more signals to one or more of the computation hardware components. The software test program generates a set of test results in response to testing the computation hardware components based on the one or more signals.

Abstract: In an embodiment, an autonomous driving vehicle (ADV) determines a LIDAR type, from of a plurality of LIDAR types, of a LIDAR unit. Responsive to determining the LIDAR type of the LIDAR unit, the ADV configures an adaptive LIDAR data processing system based on the LIDAR type. The adaptive LIDAR data processing system supports each one of the plurality of LIDAR types. In turn, responsive to configuring the adaptive LIDAR data processing system, the ADV establishes communication between the LIDAR unit and a host system using the adaptive LIDAR data processing system.

Abstract: In some implementations, a method is provided. The method includes determining a first set of data acquisition characteristics of a first sensor of an autonomous driving vehicle. The method also includes determining a second set of data acquisition characteristics of a second sensor of the autonomous driving vehicle. The method further includes synchronizing a first data acquisition time of the first sensor and a second data acquisition time of the second sensor, based on the first set of data acquisition characteristics and the second set of data acquisition characteristics. The first sensor obtains first sensor data at the first data acquisition time. The second sensor obtains second sensor data at the second data acquisition time.

Abstract: In one embodiment, an exemplary computer-implemented method of storing point cloud data in an autonomous driving vehicle can include the operations of receiving raw point cloud data from a LiDAR sensor mounted on the autonomous driving vehicle, the raw point cloud data representing cloud data points acquired in response to laser beams emitted at a given angle; retrieving configuration information of the LiDAR sensor, the configuration information including at least a number of laser lines of the LiDAR sensor. The method further includes the operations of constructing, based on the configuration information, a data structure that includes a data entry for each of the cloud data points, the data entry including multiple fields for storing attributes of the cloud data point, each field having a bit width determined based on the configuration information using a predetermined algorithm; and writing the cloud data points to a storage medium using the data structure.

Abstract: Embodiments relate to systems and methods to optimize quantization of tensors of an AI model. According to one embodiment, a system receives an AI model having one or more layers. The system receives a number of input data for offline inferencing and applies offline inferencing to the AI model based on the input data to generate offline data distributions for the AI model. The system quantizes one or more tensors of the AI model based on the offline data distributions to generate a low-bit representation AI model, where each layer of the AI model includes the one or more tensors, where the one or more tensors include the one or more tensors. In one embodiment, the system applies online inferencing using the low-bit representation AI model to generate online data distributions for a feature map, and quantizes a feature map tensor based on the online data distributions.

Abstract: A method and system is disclosed for protecting neural network models by segmenting partitions of the models into segments of pre-configured memory size, hashing the segmented models, and concatenating the hash segments. The concatenated hash segment may be further hashed, encrypted, and stored with the neural network models as an executable loadable file (ELF) in memories external to the neural network prior to the use of the models by the neural network. The models may include model weights of the inference layers and metadata. The model weights and the metadata may be hashed as separate hash segments and concatenated. Segmenting the models into segments of pre-configured memory size and hashing the segmented models offline prior to the operation of the neural network enables rapid validation of the models when the models are used in the inference layers during online operation of the neural network.

Abstract: In some implementations, a method of verifying operation of a sensor is provided. The method includes causing a sensor to obtain sensor data at a first time, wherein the sensor obtains the sensor data by emitting waves towards a detector. The method also includes determining that the detector has detected the waves at a second time. The method further includes receiving the sensor data from the sensor at a third time. The method further includes verifying operation of the sensor based on at least one of the first time, the second time, or the third time.

Abstract: A data processing system includes a host system and one or more expansion devices coupled to the host system over a bus. The host system may include one or more processors and a memory storing instructions, which when executed, cause the processors to perform autonomous driving operations to drive an autonomous driving vehicle (ADV). Each expansion device includes a switch device and one or more processing modules coupled to the switch device. Each processing module can be configured to perform at least one of the autonomous driving operations offloaded from the host system. At least one of the processing modules can be configured as a client node to perform an action in response to an instruction received from the host system. Alternatively, it can be configured as a host node to distribute a task to another client node within the expansion device. This host node in the expansion device can further cooperate with the host system via a host-to-host connection.