Abstract: A sensor unit includes a sensor interface, host interface, and pre-processing hardware. The sensor interface is coupled to a plurality of cameras configured to capture images around an autonomous driving vehicle (ADV). The host interface is coupled to a perception and planning system. The pre-processing hardware is coupled to the sensor interface to receive images from the plurality of cameras and to perform one or more pre-processing functions on the images and to transmit pre-processed images to the perception and planning system via the host interface. The perception and planning system is configured to perceive a driving environment surrounding the ADV based on the pre-processed images and to plan a path to control the ADV to navigate through the driving environment. The pre-processing functions can adjust for different calibrations and formats across the plurality of cameras.

Abstract: In some implementations, a method is provided. The method includes determining a first set of data acquisition characteristics of a first sensor of an autonomous driving vehicle. The method also includes determining a second set of data acquisition characteristics of a second sensor of the autonomous driving vehicle. The method further includes synchronizing a first data acquisition time of the first sensor and a second data acquisition time of the second sensor, based on the first set of data acquisition characteristics and the second set of data acquisition characteristics. The first sensor obtains first sensor data at the first data acquisition time. The second sensor obtains second sensor data at the second data acquisition time.

Abstract: In one embodiment, a sensor unit receives a first GPS message from a GPS sensor, where the sensor unit is coupled between sensors and a perception and planning system of an autonomous driving vehicle (ADV). The sensor unit determines a type of the first GPS message by matching a predetermined field of the first GPS message with a list of predetermined data patterns. Each of the predetermined data patterns corresponds to one of the predetermined types of GPS messages and decodes a payload of the first GPS message using a decoding algorithm associated with the type of the first GPS message.

Abstract: The disclosure describes various embodiments for online system-level validation of sensor synchronization. According to an embodiment, an exemplary method of analyzing sensor synchronization in an autonomous driving vehicle (ADV) include the operations of acquiring raw sensor data from a first sensor and a second sensor mounted on the ADV, the raw sensor data describing a target object in a surrounding environment of the ADV; and generating an accuracy map based on the raw sensor data in view of timestamps extracted from the raw sensor data. The method further includes the operations of generating a first bounding box and a second bounding box around the target object using the raw sensor data; and performing an analysis of the first and second bounding boxes and the accuracy map using a predetermined algorithm in view of one or more pre-configured sensor settings to determine whether the first sensor and the second sensor are synchronized with each other.

Abstract: Embodiments relate to systems and methods to optimize quantization of tensors of an AI model. According to one embodiment, a system receives an AI model having one or more layers. The system receives a number of input data for offline inferencing and applies offline inferencing to the AI model based on the input data to generate offline data distributions for the AI model. The system quantizes one or more tensors of the AI model based on the offline data distributions to generate a low-bit representation AI model, where each layer of the AI model includes the one or more tensors, where the one or more tensors include the one or more tensors. In one embodiment, the system applies online inferencing using the low-bit representation AI model to generate online data distributions for a feature map, and quantizes a feature map tensor based on the online data distributions.

Abstract: In one embodiment, a sensor unit to be utilized in an autonomous driving vehicle (ADV) includes a sensor interface that can be coupled to a number of sensors mounted on a number of different locations of the ADV. The sensor unit further includes a host interface that can be coupled to a host system such as a planning and control system utilized to autonomously drive the vehicle. The sensor unit further includes a number of data transfer modules corresponding to the sensors. Each of the data transfer modules can be configured to operate in one of the operating modes, dependent upon the type of the corresponding sensor. The operating modes include a low latency mode, a high bandwidth mode, and a memory mode.

Abstract: An ADV includes a method to combine data from multiple sensors. The method compresses video data from a camera to generate compressed video data. The compressed video data are segmented. The method time synchronizes each segment of the compressed video data with data from other sensors. The method then combines each segment of the compressed video data with the corresponding time-synchronized sensor data for the other sensors. In one embodiment, each segment of the compressed video data is independently decodable. In another embodiment, each segment of the compressed video data includes a compressed video unit that is prepended with a buffered portion of the compressed video data that immediately precede the compressed video unit.

Abstract: A method to perform video compression for ADV is disclosed. The method receives multiple frames of image data from multiple cameras. Metadata are appended to each frame of the image data to generate one of multiple frames of uncompressed image data as the image data are received. The frames of uncompressed image data may be stored. To compress the image data later, the method retrieves the frames of uncompressed image data, extracts the metadata from each frame of the uncompressed image data to generate one of multiple frames of processed image data. The method compresses each frame of the processed image data with the metadata extracted to generate one of multiple frames of compressed image data. The method reattaches the metadata to a corresponding frame of the compressed image data to generate one of multiple compressed image frames. The metadata supports time synchronization and error handling of the image data.

Abstract: A sensor unit includes a sensor interface and a host interface. The sensor interface can be coupled to a number of sensors mounted on various locations of the ADV. The host interface can be coupled to a host system that is configured to perceive a driving environment surrounding the ADV based on sensor data obtained from the sensors and to plan a path to autonomously drive the ADV. The sensor unit further includes a sensor processing module and a sensor control module coupled to the sensor interface, and a time module coupled to the sensor processing module and the sensor control module. The sensor processing module is configured to process sensor data received from the sensors via the sensor interface. The sensor control module is configured to control operations of the sensors. The time module is configured to generate time to synchronize timing of the operations of the sensors.

Abstract: In one embodiment, a system determines a difference in time between a local time source and a time of a GPS sensor. The system determines a max limit in difference and a max recovery increment or max recovery time interval for a smooth time source recovery. The system determines that the difference between the local time source and a time of the GPS sensor to be less than the max limit. The system plans a smooth recovery of the time source to converge the local time source to a time of the GPS sensor within the max recovery time interval. The system generates a timestamp based on the recovered time source to timestamp sensor data for a sensor unit of the ADV.

Abstract: A method and system is disclosed for protecting neural network models by segmenting partitions of the models into segments of pre-configured memory size, hashing the segmented models, and concatenating the hash segments. The concatenated hash segment may be further hashed, encrypted, and stored with the neural network models as an executable loadable file (ELF) in memories external to the neural network prior to the use of the models by the neural network. The models may include model weights of the inference layers and metadata. The model weights and the metadata may be hashed as separate hash segments and concatenated. Segmenting the models into segments of pre-configured memory size and hashing the segmented models offline prior to the operation of the neural network enables rapid validation of the models when the models are used in the inference layers during online operation of the neural network.

Abstract: In one embodiment, a system receives a number of times from a number of time sources including sensors and real-time clocks (RTCs), wherein the sensors are in communication with an autonomous driving vehicle (ADV) and the sensors include at least a GPS sensor. The system generates a difference histogram based on a time for each of the time sources for a difference between a time of the GPS sensor and a time for each of the other sensors and RTCs. The system ranks the sensors and RTCs based on the difference histogram. The system selects a time source from one of the sensors or RTCs with a least difference in time with respect to the GPS sensor. The system generates a timestamp based on the selected time source to timestamp sensor data for a sensor unit of the ADV.

Abstract: The disclosure describes various embodiments for online system-level validation of sensor synchronization. According to an embodiment, an exemplary method of analyzing sensor synchronization in an autonomous driving vehicle (ADV) include the operations of acquiring raw sensor data from a first sensor and a second sensor mounted on the ADV, the raw sensor data describing a target object in a surrounding environment of the ADV; and generating an accuracy map based on the raw sensor data in view of timestamps extracted from the raw sensor data. The method further includes the operations of generating a first bounding box and a second bounding box around the target object using the raw sensor data; and performing an analysis of the first and second bounding boxes and the accuracy map using a predetermined algorithm in view of one or more pre-configured sensor settings to determine whether the first sensor and the second sensor are synchronized with each other.

Abstract: The disclosure describes various embodiments of validating data synchronization between an active sensor and a passive sensor. According to an exemplary method of validating sensor synchronization between an active sensor and a passive sensor, a synchronization device receives a first signal from the active sensor, the first signal indicating that the active sensor has transmitted laser points to a measure board. In response to the first signal, the synchronization device transmits a second signal to the passive sensor to trigger the passive sensor to capture an image of the measure board. A synchronization validation application can perform an analysis of the image of the measure board in view of timing of the first signal and second signal to determine whether the passive sensor and the active sensor are synchronized with each other.

Abstract: In one embodiment, an exemplary computer-implemented method of storing point cloud data in an autonomous driving vehicle can include the operations of receiving raw point cloud data from a LiDAR sensor mounted on the autonomous driving vehicle, the raw point cloud data representing cloud data points acquired in response to laser beams emitted at a given angle; retrieving configuration information of the LiDAR sensor, the configuration information including at least a number of laser lines of the LiDAR sensor. The method further includes the operations of constructing, based on the configuration information, a data structure that includes a data entry for each of the cloud data points, the data entry including multiple fields for storing attributes of the cloud data point, each field having a bit width determined based on the configuration information using a predetermined algorithm; and writing the cloud data points to a storage medium using the data structure.

Abstract: Diagnosing a sensor processing unit of an autonomous driving vehicle is described. An example computer-implemented method can include transmitting an executable image of a sensor processing application from a host system to the sensor processing unit via at least one of a universal asynchronous receiver-transmitter (UART) or an Ethernet connection. The method also includes causing the sensor processing unit to execute and launch the executable image of the sensor processing application in the DRAM from the eMMC storage device. The method also includes transmitting a sequence of predetermined commands to the executed sensor processing application to perform a plurality of sensor data processing operations on sensor data obtained from a plurality of sensors or sensor simulators associated with an autonomous driving vehicle.

Abstract: A sensor unit includes a sensor interface coupled to a number of sensors and a host interface coupled to a host system utilized to autonomously drive the vehicle. The sensor unit further includes sensor control modules corresponding to the sensors. Each sensor control module includes delay time control logic, delay adjustment logic, and a trigger signal generator. The delay time control logic is to receive a pulse time adjustment (PTA) value from the host system. The delay adjustment logic is to receive a trigger time adjustment (TTA) value from the host system. The delay adjustment logic is to modify timing of at least a portion of the pulses of a pulse signal based on the PTA value and the TTA value. The trigger signal generator is to generate a trigger signal based on the modified pulse signal and to transmit the trigger signal to a corresponding sensor.

Abstract: Systems and methods are disclosed for performing manufacturing testing on an autonomous driving vehicle (ADV) sensor board. A sensor unit of the ADV includes a plurality of sensor I/O channels that provide information to the ADV perception and planning module, to navigate the ADV. An array of sensors is emulated on a sensor unit test board. The sensor unit includes a small software that manages the flow of testing the sensor unit. The sensor unit test board provides emulated sensor data for, e.g., GPS, LIDAR, RADAR, inertial measurement, one or more cameras, humidity, temperature, and pressure, and throttle, braking, and steering inputs. Each emulated sensor includes its own data validity checker to ensure that each sensor I/O channel of the sensor unit is tested.

Abstract: Flexible hardware designs for camera calibration and image pre-processing are disclosed for vehicles including autonomous driving (AD) vehicles. For one example, a sensor unit includes a sensor interface, host interface, and pre-processing hardware. The sensor interface is coupled to a plurality of cameras configured to capture images around an autonomous driving vehicle (ADV). The host interface is coupled to a perception and planning system. The pre-processing hardware is coupled to the sensor interface to receive images from the plurality of cameras and to perform one or more pre-processing functions on the images and to transmit pre-processed images to the perception and planning system via the host interface. The perception and planning system is configured to perceive a driving environment surrounding the ADV based on the pre-processed images and to plan a path to control the ADV to navigate through the driving environment.

Abstract: In one embodiment, a system receives, at a sensor unit, a global positioning system (GPS) pulse signal from a GPS sensor of the autonomous driving vehicle (ADV), where the GPS pulse signal is a RF signal transmitted by a satellite to the GPS sensor, where the sensor unit is coupled to a number of sensors mounted on the ADV to perceive a driving environment surrounding the ADV and to plan a path to autonomously drive the ADV. The system receives a first local oscillator signal from a local oscillator. The system synchronizes the first local oscillator signal to the GPS pulse signal in real-time, including modifying the first local oscillator signal based on the GPS pulse signal. The system generates a second oscillator signal based on the synchronized first local oscillator signal, where the second oscillator signal is used to provide a time to at least one of the sensors.