Abstract: Methods, systems and devices described herein implement a task-driven machine learning-based compression scheme for point cloud geometry implicit representation. The machine learning-based codec is able to be optimized for a task to achieve better compression rates by being conditioned to what the reconstructed signal will be used for. The latent representation of the point cloud or the neural network that implicitly represents the point cloud itself are able to be compressed. The methods described herein perform efficient compression of the implicit representation of a point cloud given a target task.

Abstract: A novel method of classifying point cloud data by extending the Gray-level Co-occurrence Matrix (GLCM) technique from the 2D to the sparse 3D domain is described herein. The method is able to be applied to point clouds derived from a mesh collection/meshes (such as, the Real-World Textured Things (RWTT) mesh collection). Implementations designed for multiple purposes are described herein: sampling and quantization of RWTT meshes, generation of GLCMs and corresponding texture descriptors, and the selection of potential candidate point clouds based on these extracted descriptors.

Abstract: An electronic device and method for adaptive mode selection for point cloud compression, is provided. The electronic device receives a 3D point cloud geometry and partitions the 3D point cloud geometry into a set of 3D blocks. For a 3D block of the set of 3D blocks, mode decision information is determined. The mode decision information includes class information of the 3D point cloud geometry, operational conditions associated with an encoding stage of the 3D point cloud geometry, or mode-related information associated with one or more 3D blocks of the set of 3D blocks. Based on the mode decision information, one or more modes are selected for the 3D block from a plurality of modes. Each mode corresponds to a function that is used to encode the 3D block. The 3D block is encoded based on the one or more modes.

Abstract: Described herein is a method to segment meshes into sub-meshes based on triangle properties. The triangles are first classified according to some characteristic using their respective areas. A filtering process may change the classification according to the neighboring triangles. Then connected components are generated, and neighboring connected components are merged following a certain criteria. In some embodiments, connected components that share the most amount of edges are merged. With this technique, sub-meshes can be automatically generated without any previous knowledge of the mesh generation stage.

Abstract: Ways to simplify connectivity data for patches are described herein. The patches are generated considering the high-resolution mesh information. The connectivity data is simplified at the patch level, while the geometry image is still preserved. For the connectivity simplification, only triangles inside the patch are simplified. If the border is still preserved, the reconstruction in 3D will not suffer from artifacts. The high-resolution geometry image can be used to reverse the simplification and improve the connectivity at the decoder side. Three embodiments of patch mesh simplification are described: quadric error edge collapse, border distance edge collapse, and border triangles only.

Abstract: An electronic device and method for a variable rate compression of a point cloud geometry is provided. The electronic device stores a set of RD operation points and coding modes associated with the set of RD operation points. The electronic device receives a 3D point cloud geometry and partitions the 3D geometry into a set of blocks. After the partition, the electronic device selects a block and computes a set of loss values associated with one or more compression metrics. Such loss values correspond to a set of coding modes associated with at least a subset of the set of RD operation points. From the set of coding modes, the electronic device selects a coding mode for which a loss value of the set of loss values is below a loss threshold for that coding mode. Thereafter, the electronic device encodes the block based on the coding mode.

Abstract: An attribute prediction and compensation scheme for geometry-based dynamic point cloud compression is described herein. A combination of multiple reference frames are able to be used as a predictor for current frames. The method described herein improves efficiency and accuracy.

Abstract: An electronic device and method for multiscale inter-prediction for dynamic point cloud compression is provided. The electronic device receives a set of reference point cloud frames and a current point cloud frame. The electronic device generates reference frame data comprising a feature set for each reference point cloud frame and a first set of features for the current point cloud frame. The electronic device predicts a second set of features for the current point cloud frame, using a first neural network predictor, based on the reference frame data. The electronic device computes a set of residual features based on the first set of features and the second set of features. The electronic device generates a set of quantized residual features based on the set of residual features and a bitstream of encoded point cloud data for the current 3D point cloud frame based on the set of quantized residual features.

Abstract: An electronic device and method for detection and indication of geometry reconstruction artifacts is provided. The electronic device acquires a reference point cloud, encodes the reference point cloud to generate encoded point cloud data, and decodes the encoded point cloud data to generate a test point cloud. The electronic device further generates a first local density map and a second local density map for points of the reference point cloud and the test point cloud, respectively. The electronic device generates a final density map based on the first local density map and the second local density map, and further generates supplementary information based on the final density map. The supplementary information includes missing points data corresponding to regions of the test point cloud that include artifacts such as holes or includes descriptors for the regions that include the artifacts. The electronic device signals the supplementary information to a decoder.

Abstract: An electronic device and method for using a rate adaptive codec for dynamic point cloud compression is provided. The electronic device receives a point cloud frame (PCF) sequence that includes a set of reference PCFs and a current PCF that is to be encoded. The electronic device generates 3D feature data based on application of a neural network-based encoder on at least one of the set of reference PCFs and the current PCF. The electronic device selects from an encoding pipeline, an encoding path as an I-frame encoding path or a P-frame encoding path for the current PCF. The selected encoding path extends from an output of the neural network-based encoder and includes a first sequence of transformation operations that is applicable on the 3D feature data. The electronic device generates a bit-stream of coded PCF based on application of the first sequence of transformation operations on the 3D feature data.

Abstract: A new SEI message for the V-DMC standard is described herein, the zippering SEI. The zippering SEI message can be used by the decoder for the mesh reconstruction, where in the case of multiple sub-meshes, the zippering SEI provides ways to reduce common artifacts caused by independent sub-mesh encoding, such as holes and cracks on the mesh surface.

Abstract: An electronic device and method for multiscale inter-prediction for dynamic point cloud compression is provided. The electronic device receives a set of reference point cloud frames and a current point cloud frame. The electronic device generates reference frame data comprising a feature set for each reference point cloud frame and a first set of features for the current point cloud frame. The electronic device predicts a second set of features for the current point cloud frame, using a first neural network predictor, based on the reference frame data. The electronic device computes a set of residual features based on the first set of features and the second set of features. The electronic device generates a set of quantized residual features based on the set of residual features and a bitstream of encoded point cloud data for the current 3D point cloud frame based on the set of quantized residual features.

Abstract: An artificial intelligence (AI) based method for calibration of a gaming ecosystem and to maximize satisfaction of a player involved in a gaming session such as single-player gaming session, multi-player online gaming session, and a videogame recommendation system is provided. The system detects the gaming session that includes an execution of a video game for a first gameplay and acquires sensor data associated with a player involved in the first gameplay. The sensor data corresponds to a duration of the detected gaming session. The system determines one or more indicators of a dissatisfaction of the player with the first gameplay based on application of one or more Artificial Intelligence (AI) models on the sensor data. The system controls the execution of the video game to modify one or more aspects associated with the first gameplay or a second gameplay of the video game that is different from the first gameplay.

Abstract: An electronic device and method for estimation of density distortion metric for processing of point cloud geometry is provided. The electronic device acquires a reference point cloud, encodes the reference point cloud to generate encoded point cloud data, and decodes the encoded point cloud data to generate a test point cloud. The electronic device further generates a first local density map representing local density values at points of the reference point cloud. The electronic device determines locations in the test point cloud corresponding to locations of the points. The electronic device generates a second local density map representing local density values at the determined locations. The electronic device computes a value of density distortion metric based on the first local density map and the second local density map. The electronic device controls a display device to render a reconstruction quality of the test point cloud based on the computed value.

Abstract: A method of compression of 3D mesh data using projections of mesh surface data and video representation of connectivity data is described herein. The method utilizes 3D surface patches to represent a set of connected triangles on a mesh surface. The projected surface data is stored in patches (a mesh patch) that is encoded in atlas data. The connectivity of the mesh, that is, the vertices and the triangles of the surface patch, are encoded using video-based compression techniques. The data is encapsulated in a new video component named vertex video data, and the disclosed structure allows for progressive mesh coding by separating sets of vertices in layers, and creating levels of detail for the mesh connectivity. This approach extends the functionality of the V3C (volumetric video-based) standard, currently being used for coding of point cloud and multiview plus depth content.

Abstract: Depth image generation is improved by more efficient encoding using video codecs. The mapping of the depth to the luma channel is performed by not using all bits available, and with the remaining bits, a depth scaling factor is generated and incorporated into the bilinear interpolation algorithm used during rasterization. A normal filtering procedure is described, where the positions of vertices are adjusted according to the normal estimated from the surface pixels. After decoding the depth image, the pixels related to the surface of a triangle are collected and used to estimate a plane and the normal of the plane. The normal is compared to the normal obtained from the plane defined by the three vertices of the triangle. If there is no match, the positions of the vertices are adjusted to match the estimated normal from the pixels' surfaces. The adjustment can follow an iterative minimization process.

Abstract: New syntax elements are used to extend patch types, and the syntax is added to the V3C standard. The new syntax defines patches that encode meshes by projecting connected triangles to a 2D surface, patches that encode triangles or triangle strips without any projection, or patches that are tracked over time and are encoded by projecting connected triangles to a 2D surface. Furthermore, the syntax allows for different ways of coding the mesh-specific information. For instance, the syntax enables three different encoding methods for the vertex position: explicit (directly added to the atlas stream), embedded on video data (occupancy map data), or encoded using an external mesh encoder.

Abstract: Ways to post-process a decoded mesh and modify received triangles per patch to improve the mesh geometry are described herein. Since the transmitted geometry contains the high-resolution surface information, the information is able to be applied to the mesh reconstruction operation to generate triangles that are aligned with the original surface. Methods include generating new triangles by splitting the received triangles' edges according to their size, by inserting new vertices at the triangle's centroids, by splitting the vertices, and by performing marching cubes in surfaces defined by the geometry images.

Abstract: Ways to improve mesh reconstruction by modifying the position of vertices at the border of patches to make sure that neighboring patches do not have a gap between them, also known as zippering, are described herein. Six different methods to implement the post-processing operation, as well as syntax elements and semantics for transmission of the filter parameters, are disclosed. A hierarchical method indicate the geometry distortion that can generate gaps between patches. The value per frame, or per patch, or per boundary object is sent. The number of bits to encode the values is also dependent on the previous geometry distortion. A method sends index matches instead of geometry distortion. The matching index is sent per boundary vertex, but a method to send only one index of the pair is implemented as well.

Abstract: The connectivity information and mapping information of a mesh surface patch are able to be encoded after projection to 2D. Regarding the connectivity information, the projection operation does not change the connection between vertices, so the same list of connected vertices are able to be carried in the atlas data. Similarly, the mapping information does not change after projection and is able to be carried in the atlas data. Two methods are disclosed for encoding the connectivity and mapping information. For the connectivity information, a video-based method uses neighboring color coding. For mapping coordinates, a method uses the projected vertex position. The connectivity and mapping are also able to be processed by an external mesh encoder. Newly proposed mapping information is able to be taken advantage of to perform temporal compression.