Abstract: An ADV includes a method to combine data from multiple sensors. The method compresses video data from a camera to generate compressed video data. The compressed video data are segmented. The method time synchronizes each segment of the compressed video data with data from other sensors. The method then combines each segment of the compressed video data with the corresponding time-synchronized sensor data for the other sensors. In one embodiment, each segment of the compressed video data is independently decodable. In another embodiment, each segment of the compressed video data includes a compressed video unit that is prepended with a buffered portion of the compressed video data that immediately precede the compressed video unit. The length of the compressed video unit is smaller than the length of the independently decodable segment to offer finer granularity in time synchronizing the compressed video data with the other sensor data with a tradeoff.

Abstract: A method, apparatus, and system for timing synchronization between multiple computing nodes in an autonomous vehicle host system is disclosed. Timing of a first computing node of an autonomous vehicle host system is calibrated based on an external time source. A first timing message is transmitted from the first computing node to a second computing node of the autonomous vehicle host system via a first communication channel between the first computing node and the second computing node. Timing of the second computing node is calibrated based on the first timing message, wherein immediately subsequent to the calibration of timing of the second computing node, timing of the first computing node and of the second computing node is synchronized.

Abstract: In one embodiment, a computer-implemented method of performing a secure boot operation in an autonomous driving vehicle includes reading a first marker from a storage device in which the storage device includes a plurality of partitions and at least the first marker. The plurality of partitions includes a first partition including stored software, the first marker associated with the first partition, and wherein the first marker includes a unique identifier and an authentication code. The method further includes determining if the read first marker associated with the first partition is valid during a boot-up operation and executing the stored software in the first partition if the read first marker is determined valid.

Abstract: Systems and methods are disclosed for performing manufacturing testing on an autonomous driving vehicle (ADV) sensor board. A sensor unit of the ADV includes a plurality of sensor I/O channels that provide information to the ADV perception and planning module, to navigate the ADV. An array of sensors is emulated on a sensor unit test board. The sensor unit includes a small software that manages the flow of testing the sensor unit. The sensor unit test board provides emulated sensor data for, e.g., GPS, LIDAR, RADAR, inertial measurement, one or more cameras, humidity, temperature, and pressure, and throttle, braking, and steering inputs. Each emulated sensor includes its own data validity checker to ensure that each sensor I/O channel of the sensor unit is tested. The sensor unit test board can include an LED for each I/O channel that indicates a pass/fail status of the test for the I/O channel.

Abstract: In one embodiment, a sensor unit to be utilized in an autonomous driving vehicle (ADV) includes a sensor interface that can be coupled to a number of sensors mounted on a number of different locations of the ADV. The sensor unit further includes a host interface that can be coupled to a host system such as a planning and control system utilized to autonomously drive the vehicle. The sensor unit further includes a number of data transfer modules corresponding to the sensors. Each of the data transfer modules can be configured to operate in one of the operating modes, dependent upon the type of the corresponding sensor. The operating modes include a low latency mode, a high bandwidth mode, and a memory mode.

Abstract: A method to perform video compression for ADV is disclosed. The method receives multiple frames of image data from multiple cameras. Metadata are appended to each frame of the image data to generate one of multiple frames of uncompressed image data as the image data are received. The frames of uncompressed image data may be stored. To compress the image data later, the method retrieves the frames of uncompressed image data, extracts the metadata from each frame of the uncompressed image data to generate one of multiple frames of processed image data. The method compresses each frame of the processed image data with the metadata extracted to generate one of multiple frames of compressed image data. The method reattaches the metadata to a corresponding frame of the compressed image data to generate one of multiple compressed image frames. The metadata supports time synchronization and error handling of the image data.

Abstract: The disclosure describes various embodiments of validating data synchronization between an active sensor and a passive sensor. According to an exemplary method of validating sensor synchronization between an active sensor and a passive sensor, a synchronization device receives a first signal from the active sensor, the first signal indicating that the active sensor has transmitted laser points to a measure board. In response to the first signal, the synchronization device transmits a second signal to the passive sensor to trigger the passive sensor to capture an image of the measure board. A synchronization validation application can perform an analysis of the image of the measure board in view of timing of the first signal and second signal to determine whether the passive sensor and the active sensor are synchronized with each other.

Abstract: Diagnosing a sensor processing unit of an autonomous driving vehicle is described. An example computer-implemented method can include transmitting an executable image of a sensor processing application from a host system to the sensor processing unit via at least one of a universal asynchronous receiver-transmitter (UART) or an Ethernet connection. The method also includes causing the sensor processing unit to execute and launch the executable image of the sensor processing application in the DRAM from the eMMC storage device. The method also includes transmitting a sequence of predetermined commands to the executed sensor processing application to perform a plurality of sensor data processing operations on sensor data obtained from a plurality of sensors or sensor simulators associated with an autonomous driving vehicle.

Abstract: In some implementations, a method of verifying operation of a sensor is provided. The method includes causing a sensor to obtain sensor data at a first time, wherein the sensor obtains the sensor data by emitting waves towards a detector. The method also includes determining that the detector has detected the waves at a second time. The method further includes receiving the sensor data from the sensor at a third time. The method further includes verifying operation of the sensor based on at least one of the first time, the second time, or the third time.

Abstract: A sensor unit includes a sensor interface, host interface, and pre-processing hardware. The sensor interface is coupled to a plurality of cameras configured to capture images around an autonomous driving vehicle (ADV). The host interface is coupled to a perception and planning system. The pre-processing hardware is coupled to the sensor interface to receive images from the plurality of cameras and to perform one or more pre-processing functions on the images and to transmit pre-processed images to the perception and planning system via the host interface. The perception and planning system is configured to perceive a driving environment surrounding the ADV based on the pre-processed images and to plan a path to control the ADV to navigate through the driving environment. The pre-processing functions can adjust for different calibrations and formats across the plurality of cameras.

Abstract: In some implementations, a method is provided. The method includes determining a first set of data acquisition characteristics of a first sensor of an autonomous driving vehicle. The method also includes determining a second set of data acquisition characteristics of a second sensor of the autonomous driving vehicle. The method further includes synchronizing a first data acquisition time of the first sensor and a second data acquisition time of the second sensor, based on the first set of data acquisition characteristics and the second set of data acquisition characteristics. The first sensor obtains first sensor data at the first data acquisition time. The second sensor obtains second sensor data at the second data acquisition time.

Abstract: In one embodiment, a sensor unit receives a first GPS message from a GPS sensor, where the sensor unit is coupled between sensors and a perception and planning system of an autonomous driving vehicle (ADV). The sensor unit determines a type of the first GPS message by matching a predetermined field of the first GPS message with a list of predetermined data patterns. Each of the predetermined data patterns corresponds to one of the predetermined types of GPS messages and decodes a payload of the first GPS message using a decoding algorithm associated with the type of the first GPS message.

Abstract: The disclosure describes various embodiments for online system-level validation of sensor synchronization. According to an embodiment, an exemplary method of analyzing sensor synchronization in an autonomous driving vehicle (ADV) include the operations of acquiring raw sensor data from a first sensor and a second sensor mounted on the ADV, the raw sensor data describing a target object in a surrounding environment of the ADV; and generating an accuracy map based on the raw sensor data in view of timestamps extracted from the raw sensor data. The method further includes the operations of generating a first bounding box and a second bounding box around the target object using the raw sensor data; and performing an analysis of the first and second bounding boxes and the accuracy map using a predetermined algorithm in view of one or more pre-configured sensor settings to determine whether the first sensor and the second sensor are synchronized with each other.

Abstract: Embodiments relate to systems and methods to optimize quantization of tensors of an AI model. According to one embodiment, a system receives an AI model having one or more layers. The system receives a number of input data for offline inferencing and applies offline inferencing to the AI model based on the input data to generate offline data distributions for the AI model. The system quantizes one or more tensors of the AI model based on the offline data distributions to generate a low-bit representation AI model, where each layer of the AI model includes the one or more tensors, where the one or more tensors include the one or more tensors. In one embodiment, the system applies online inferencing using the low-bit representation AI model to generate online data distributions for a feature map, and quantizes a feature map tensor based on the online data distributions.

Abstract: In one embodiment, a sensor unit to be utilized in an autonomous driving vehicle (ADV) includes a sensor interface that can be coupled to a number of sensors mounted on a number of different locations of the ADV. The sensor unit further includes a host interface that can be coupled to a host system such as a planning and control system utilized to autonomously drive the vehicle. The sensor unit further includes a number of data transfer modules corresponding to the sensors. Each of the data transfer modules can be configured to operate in one of the operating modes, dependent upon the type of the corresponding sensor. The operating modes include a low latency mode, a high bandwidth mode, and a memory mode.

Abstract: An ADV includes a method to combine data from multiple sensors. The method compresses video data from a camera to generate compressed video data. The compressed video data are segmented. The method time synchronizes each segment of the compressed video data with data from other sensors. The method then combines each segment of the compressed video data with the corresponding time-synchronized sensor data for the other sensors. In one embodiment, each segment of the compressed video data is independently decodable. In another embodiment, each segment of the compressed video data includes a compressed video unit that is prepended with a buffered portion of the compressed video data that immediately precede the compressed video unit.

Abstract: A method to perform video compression for ADV is disclosed. The method receives multiple frames of image data from multiple cameras. Metadata are appended to each frame of the image data to generate one of multiple frames of uncompressed image data as the image data are received. The frames of uncompressed image data may be stored. To compress the image data later, the method retrieves the frames of uncompressed image data, extracts the metadata from each frame of the uncompressed image data to generate one of multiple frames of processed image data. The method compresses each frame of the processed image data with the metadata extracted to generate one of multiple frames of compressed image data. The method reattaches the metadata to a corresponding frame of the compressed image data to generate one of multiple compressed image frames. The metadata supports time synchronization and error handling of the image data.

Abstract: A sensor unit includes a sensor interface and a host interface. The sensor interface can be coupled to a number of sensors mounted on various locations of the ADV. The host interface can be coupled to a host system that is configured to perceive a driving environment surrounding the ADV based on sensor data obtained from the sensors and to plan a path to autonomously drive the ADV. The sensor unit further includes a sensor processing module and a sensor control module coupled to the sensor interface, and a time module coupled to the sensor processing module and the sensor control module. The sensor processing module is configured to process sensor data received from the sensors via the sensor interface. The sensor control module is configured to control operations of the sensors. The time module is configured to generate time to synchronize timing of the operations of the sensors.

Abstract: In one embodiment, a system determines a difference in time between a local time source and a time of a GPS sensor. The system determines a max limit in difference and a max recovery increment or max recovery time interval for a smooth time source recovery. The system determines that the difference between the local time source and a time of the GPS sensor to be less than the max limit. The system plans a smooth recovery of the time source to converge the local time source to a time of the GPS sensor within the max recovery time interval. The system generates a timestamp based on the recovered time source to timestamp sensor data for a sensor unit of the ADV.

Abstract: A method and system is disclosed for protecting neural network models by segmenting partitions of the models into segments of pre-configured memory size, hashing the segmented models, and concatenating the hash segments. The concatenated hash segment may be further hashed, encrypted, and stored with the neural network models as an executable loadable file (ELF) in memories external to the neural network prior to the use of the models by the neural network. The models may include model weights of the inference layers and metadata. The model weights and the metadata may be hashed as separate hash segments and concatenated. Segmenting the models into segments of pre-configured memory size and hashing the segmented models offline prior to the operation of the neural network enables rapid validation of the models when the models are used in the inference layers during online operation of the neural network.