Abstract: In one implementation, we propose an unsupervised point cloud primitive learning method based on the principle of analysis by synthesis. In one example, the method uses a partitioning network and a point cloud autoencoder. The partitioning network partitions an input point cloud into a list of chunks. For each chunk, an encoder network of the autoencoder performs analysis to output a codeword in a feature space, and a decoder network performs synthesis to reconstruct the point cloud chunk. The reconstructed chunks are merged to output a fully reconstructed point cloud frame. By end-to-end training to minimize the mismatch between the original point cloud and the reconstructed point cloud, the autoencoder discovers primitive shapes in the point cloud data. During the network training, the parameters of the partitioning network and the autoencoder are updated. The trained modules can be applied to different applications, including segmentation, detection, and compression.

Abstract: In one implementation, we propose a lossy point cloud compression scheme to encode point cloud geometry with deep neural networks. The encoder first encodes a coarser version of the input point cloud as a bitstream. Then it represents the residual (fine geometry details) of the input point cloud as pointwise features of the encoded coarser point cloud, followed by encoding the features as the second bitstream. On the decoder side, the coarser point cloud is firstly decoded from the first bitstream. Then its pointwise features are decoded. In the end, the residual is decoded from the pointwise features and added back to the coarser point cloud, leading to a high-quality decoded point cloud. The encoding and/or decoding of the features can be further augmented with feature aggregation, such as transformer blocks.

Abstract: Systems, methods, and instrumentalities are disclosed for coordinate refinement and/or up-sampling from quantized point cloud reconstruction. In examples, point-based coordinate refinement may be provided. An after-decoder point cloud refinement module may include one or more of the following. The module may include accessing a decoded quantized version of a point cloud. The module may include accessing and/or fetching point(s) within a neighborhood area of each of the point(s). A feature may be computed using a point-based neural network module, for example, based on the three-dimensional (3D) (e.g., or KD) location(s) of the fetched points, e.g., that summarizes the details (e.g., intricate details). A refinement offset for the current, point may be predicted based on the comprehensive featuring using a fully connected (FC) module.

Abstract: In one implementation, we propose a hybrid architecture to compress and decompress a point cloud. In particular, a first decoding block is for the most significant bits, typically coded by a tree-based coding method. A second decoding block is for the middle-range of bits, typically coded by a voxel-based method. A third decoding block is for the least significant bits, typically coded by a point-based method. For example, the decoder configures the decoder's network according to the total number of bits and the bit partitioning positions; decodes a coarse point cloud and its associated point-wise features using a tree-based decoding block; upsamples the coarse point cloud to a denser one and updates the point-wise features using a voxel-based decoding block; and refines the precision of the coordinates of the dense but low bit depth point cloud to high bit depth point cloud using a point-based decoding block.

Abstract: In one implementation, we improve the binary voxel-based octree coding method, via a proposed state summarization module for context modeling. Given a current voxel to be encoded or decoded, instead of directly estimating its occupancy probability based on the associated binary occupancy context, a proposed state summarization module is applied to convert the original binary context to a summarized representation. Under the summarized representation, the estimation of the occupancy probability becomes more affordable and effective. In particular, density-based state summarization, pattern-based, learning-based state summarization, and learning-based state summarization methods are provided.

Abstract: Methods and apparatuses for decoding and encoding point cloud data are described herein. A method may include accessing point cloud data compressed based on a tree structure. The method may further comprise fetching points in a neighborhood associated with a current node of the tree structure, and computing a feature using a point-based neural network module, based on three-dimensional (3D) locations of the fetched points. The method may include predicting, using a neural network module, an occupancy symbol distribution for the current node based on the feature, and determining the occupancy for the current node from the encoded bitstream and the predicted occupancy symbol distribution. The method may include computing another feature using a convolution-based neural network module, based on a voxelized version of the fetched points, and fusing the feature and the another feature with one or more known features of a current node to compose a comprehensive feature.

Abstract: In one implementation, we propose the UnfoldingOperator, which unfolds/flattens an unorganized input 3D point cloud onto a regular 2D grid. Given an input point cloud, an input 2D grid and the reconstructed point cloud produced by the FoldingNet, our proposal maps the input point cloud onto the 2D grid based on the reconstructed point cloud, leading to a 3-channel image. Alternatively, instead of using an image alone to represent a point cloud, the point cloud is decomposed into a codeword and a 3-channel residual image. This residual image is obtained by subtracting the reconstructed point cloud from the original input. The proposed UnfoldingOperator can be applied to point cloud compression, leading to a corresponding compression system that we call UnfoldingCompression. The UnfoldingCompression can work with the TearingCompression, where we can adaptively choose whether to use the UnfoldingCompression or TearingCompression.

Abstract: In one implementation, a learnable transformation TearingTransform over 3D point cloud data is proposed. The TearingTransform could decompose point clouds into two channels: a low rank channel and a sparse channel. The low rank channel corresponds to a codeword representing a rough shape of a point cloud. The sparse channel appears as an image-like data representing residual information that can refine the reconstructed point locations. In an encoder based on TearingTransform, a PN module is used to generate the codeword from the input point cloud; a FN module is used to reconstruct a preliminary point cloud from the codeword and an initial grid image; and a TN module modifies the initial grid image to generate an adjusted grid image. The codeword and the adjusted grid image are compressed. At the decoder, the point cloud can be reconstructed based on the decompressed codeword and adjusted grid image.

Abstract: A method, apparatus or system for processing point cloud information can involve a learned deep entropy model over octrees for lossless compression/decompression of 3D point cloud data, wherein self-supervised compression/decompression involves an adaptive entropy coder operating on a tree-structured conditional entropy model and utilizing information from the local neighborhood as well as the global topology from the tree structure.

Abstract: A method for adaptively abstracting a point cloud includes initializing a set of primitives associated with a query shape and a set of query parameters. For each primitive a local point set is accessed using the set of query parameters and the query shape associated with the primitive. For each local point set, using a first neural network, a descriptor vector comprising a sub-vector for a primitive update and a sub-vector for a local descriptor is determined. The set of primitives is updated based on the descriptor vector for each local point set.

Abstract: A method includes generating, using a neural network, a point-level feature vector for each point of a point cloud and a set-level feature vector for the point cloud. A representative position is generated based on the point-level feature vectors and on the set-level feature vector. The representative position and the set-level feature vector is output as a set descriptor.

Abstract: Method, apparatus and system implemented by a neural network-based decoder (NNBD) are disclosed. In one method, the NNBD may obtain or receive a codeword, as a descriptor of an input data representation. A first neural network module may determine, based on at least the codeword and an initial graph, a preliminary reconstruction of the input data representation. The NNBD may determine, based on at least the preliminary reconstruction and the codeword, a modified graph. The first neural network module may determine, based on at least the codeword and the modified graph, a refined reconstruction of the input data representation. The modified graph may indicate topology information associated with the input data representation.

Abstract: Systems and methods are described for refining first point cloud data using at least second point cloud data and one or more sets of quantizer shifts. An example point cloud decoding method includes obtaining data representing at least a first point cloud and a second point cloud; obtaining information identifying at least a first set of quantizer shifts associated with the first point cloud; and obtaining refined point cloud data based on at least the first point cloud, the first set of quantizer shifts, and the second point cloud. The obtaining of the refined point cloud data may include performing a subtraction based on at least the first set of quantizer shifts. Corresponding encoding systems and methods are also described.

Abstract: An image processing method and apparatus, an electronic device, and a storage medium are provided. The method includes: obtaining an infrared image and a color image for the same object in a predetermined scenario, and a first depth map corresponding to the color image; obtaining a first optical flow between the color image and the infrared image; and performing first optimization processing on the first depth map by using the first optical flow to obtain an optimized second depth map corresponding to the color image.

Abstract: An image processing method and apparatus, an electronic device, and a storage medium are provided. The method includes: obtaining an infrared image and a color image for the same object in a predetermined scenario, and a first depth map corresponding to the color image; obtaining a first optical flow between the color image and the infrared image; and performing first optimization processing on the first depth map by using the first optical flow to obtain an optimized second depth map corresponding to the color image.