Abstract: Devices, systems, methods, and media are disclosed for performing an object detection task comprising: obtaining a semantic segmentation map representing a real-world space, the semantic segmentation map including an array of elements that each represent a respective location in the real-world space and are assigned a respective element classification label; clustering groups of the elements based on the assigned respective element classification labels to identify at least a first cluster of elements that have each been assigned the same respective element classification label; generating, based on a location of the first cluster within the semantic segmentation map, at least one anchor that defines a respective probable object location of a first dynamic object; and generating, based on the semantic segmentation map and the at least one anchor, a respective bounding box and object instance classification label for the first dynamic object.

Abstract: A method and processing unit for training a neural network to selectively quantize weights of a filter of the neural network as either binary weights or ternary weights. A plurality of training iterations a performed that each comprise: quantizing a set of real-valued weights of a filter to generate a corresponding set of quantized weights; generating an output feature tensor based on matrix multiplication of an input feature tensor and the set of quantized weights; computing, based on the output feature tensor, a loss based on a regularization function that is configured to move the loss towards a minimum value when either: (i) the quantized weights move towards binary weights, or (ii) the quantized weights move towards a ternary weights; computing a gradient with an objective of minimizing the loss; updating the real-valued weights based on the computed gradient.

Abstract: Systems, methods and apparatus for sematic segmentation of 3D point clouds using deep neural networks. The deep neural network generally has two primary subsystems: a multi-branch cascaded subnetwork that includes an encoder and a decoder, and is configured to receive a sparse 3D point cloud, and capture and fuse spatial feature information in the sparse 3D point cloud at multiple scales and multi hierarchical levels; and a spatial feature transformer subnetwork that is configured to transform the cascaded features generated by the multi-branch cascaded subnetwork and fuse these scaled features using a shared decoder attention framework to assist in the prediction of sematic classes for the sparse 3D point cloud.

Abstract: Systems and methods for panoptic segmentation of a point cloud are provided. A point cloud is projected into a range image. Features are extracted from the range image and generating a feature map from the extracted features. The feature map is downsampled and the features are scaled during downsampling using local geometry. Features are extracted from the downsampled feature map. The point cloud is semantically segmented at least partially based on the features extracted. Instances in the point cloud are segmented at least partially based on the features extracted.

Abstract: Method and system for processing a point cloud frame representing a real-world scene that includes one or more objects, including assigning data-element-level classification labels to data elements that each respectively represent one or more points included in the point cloud frame, estimating an approximate position of a first object instance represented in the point cloud frame, assigning an object-instance-level classification label to the first object instance, selecting, for the first object instance, a subgroup of the data elements based on the approximate position, selecting from the subgroup a first cluster of data elements that have assigned data-element-level classification labels that match the object-instance-level classification label assigned to the first object instance, and outputting an object instance list that indicates, for the first object instance, the first cluster of data elements, and the object-instance-level classification label assigned to the first object instance.

Abstract: Devices, systems, methods, and media are disclosed for performing an object detection task comprising: obtaining a semantic segmentation map representing a real-world space, the semantic segmentation map including an array of elements that each represent a respective location in the real-world space and are assigned a respective element classification label; clustering groups of the elements based on the assigned respective element classification labels to identify at least a first cluster of elements that have each been assigned the same respective element classification label; generating, based on a location of the first cluster within the semantic segmentation map, at least one anchor that defines a respective probable object location of a first dynamic object; and generating, based on the semantic segmentation map and the at least one anchor, a respective bounding box and object instance classification label for the first dynamic object.

Abstract: In methods and systems for graph-based panoptic segmentation of point clouds, points of a point cloud are received with a semantic label from a first category. Further, a plurality of unified cluster feature vectors from a second category are received, each being extracted from a cluster of points in the point cloud. Nodes of a constructed graph represent the unified feature vectors, and edges indicate the relationship between pairs of nodes. The edges are represented as an adjacency matrix indicating the existence or absence of an edge between pairs of nodes. A graph convolutional neural network uses the graph to predict an instance label for each node or an attribute for each edge, wherein the attribute of each edge is used for assigning the instance label to each node.

Abstract: Method and system for computing a total variation loss for use in backpropagation during training a neural network which individually classifies data points, comprising: predicting, using a neural network, a respective label for each data point in a set of input data points; determining a variation indicator that indicates a variance between: (i) smoothness of the predicted labels among neighboring data points and (ii) smoothness of the ground truth labels among the same neighboring data points; and computing the total variation loss based on the variation indicator.

Abstract: Methods and systems for computing a surface normal vector for a point cloud are disclosed. A sparse range map is generated by projecting the point cloud. A vertical kernel and a horizontal kernel are used to generate a vertical kernel range map and a horizontal kernel range map from the sparse range map. To complete an empty entry in the sparse range map, a guidance range map is used compute a vertical gradient and a horizontal gradient. The empty entry is filled using a corresponding entry in the vertical kernel range map or in the horizontal kernel range map, based on a comparison between the vertical gradient and the horizontal gradient. A surface normal vector is computed using the completed dense range map.

Abstract: A method and system for clustering-based panoptic segmentation of point clouds and a method of training the same are provided. Features of a point cloud that includes a plurality of points are extracted. Clusters of the plurality of points corresponding to objects from the features of the point cloud frame are identified. A subset of the plurality of points is selectively shifted using the features and the clusters of the plurality of points via a neural network that is trained to recognize a subset of points of objects that are closer to points of other objects than a distance between centroids of the corresponding objects and shift the subset of points away from the other objects.

Abstract: Systems, methods and apparatus for sematic segmentation of 3D point clouds using deep neural networks. The deep neural network generally has two primary subsystems: a multi-branch cascaded subnetwork that includes an encoder and a decoder, and is configured to receive a sparse 3D point cloud, and capture and fuse spatial feature information in the sparse 3D point cloud at multiple scales and multi hierarchical levels; and a spatial feature transformer subnetwork that is configured to transform the cascaded features generated by the multi-branch cascaded subnetwork and fuse these scaled features using a shared decoder attention framework to assist in the prediction of sematic classes for the sparse 3D point cloud.

Abstract: Systems and methods are disclosed for processing sparse tensors using a trained neural network model. An input sparse tensor may represent a sparse input point cloud. The input sparse tensor is processed using an encoder stage having a series of one or more encoder blocks, wherein each encoder block includes a sparse convolution layer, a sparse intra-channel attention module, a sparse inter-channel attention module, and a sparse residual tower module. Output from the encoder stage is processed using a decoder stage having a series of one or more decoder blocks, wherein each decoder block includes a sparse transpose convolution layer, a sparse inter-channel attention module, and a sparse residual tower module. The output of the decoder stage is an output sparse tensor representing a sparse labeled output point cloud.

Abstract: Methods and systems for graph-based panoptic segmentation of point clouds are described herein. The methods receive points of a point cloud with a semantic label from a first category. Further, a plurality of unified cluster feature vectors from a second category are received. Each unified cluster feature vector is extracted from a cluster of points in the point cloud. A graph comprising nodes and edges is constructed from the plurality of unified cluster feature vectors. Each node of the graph is the unified feature vector, and each edge of the graph indicates the relationship between every two nodes of the graph. The edges of the graph are represented as an adjacency matrix, wherein the adjacency matrix indicates the existence, or the lack of existence, of an edge between every two nodes.

Abstract: A method and processing unit for training a neural network to selectively quantize weights of a filter of the neural network as either binary weights or ternary weights. A plurality of training iterations a performed that each comprise: quantizing a set of real-valued weights of a filter to generate a corresponding set of quantized weights; generating an output feature tensor based on matrix multiplication of an input feature tensor and the set of quantized weights; computing, based on the output feature tensor, a loss based on a regularization function that is configured to move the loss towards a minimum value when either: (i) the quantized weights move towards binary weights, or (ii) the quantized weights move towards a ternary weights; computing a gradient with an objective of minimizing the loss; updating the real-valued weights based on the computed gradient.