Abstract: Methods, systems and devices described herein implement a task-driven machine learning-based compression scheme for point cloud geometry implicit representation. The machine learning-based codec is able to be optimized for a task to achieve better compression rates by being conditioned to what the reconstructed signal will be used for. The latent representation of the point cloud or the neural network that implicitly represents the point cloud itself are able to be compressed. The methods described herein perform efficient compression of the implicit representation of a point cloud given a target task.

Abstract: A novel method of classifying point cloud data by extending the Gray-level Co-occurrence Matrix (GLCM) technique from the 2D to the sparse 3D domain is described herein. The method is able to be applied to point clouds derived from a mesh collection/meshes (such as, the Real-World Textured Things (RWTT) mesh collection). Implementations designed for multiple purposes are described herein: sampling and quantization of RWTT meshes, generation of GLCMs and corresponding texture descriptors, and the selection of potential candidate point clouds based on these extracted descriptors.

Abstract: An electronic device and method for adaptive mode selection for point cloud compression, is provided. The electronic device receives a 3D point cloud geometry and partitions the 3D point cloud geometry into a set of 3D blocks. For a 3D block of the set of 3D blocks, mode decision information is determined. The mode decision information includes class information of the 3D point cloud geometry, operational conditions associated with an encoding stage of the 3D point cloud geometry, or mode-related information associated with one or more 3D blocks of the set of 3D blocks. Based on the mode decision information, one or more modes are selected for the 3D block from a plurality of modes. Each mode corresponds to a function that is used to encode the 3D block. The 3D block is encoded based on the one or more modes.

Abstract: Described herein is a method to segment meshes into sub-meshes based on triangle properties. The triangles are first classified according to some characteristic using their respective areas. A filtering process may change the classification according to the neighboring triangles. Then connected components are generated, and neighboring connected components are merged following a certain criteria. In some embodiments, connected components that share the most amount of edges are merged. With this technique, sub-meshes can be automatically generated without any previous knowledge of the mesh generation stage.

Abstract: Ways to simplify connectivity data for patches are described herein. The patches are generated considering the high-resolution mesh information. The connectivity data is simplified at the patch level, while the geometry image is still preserved. For the connectivity simplification, only triangles inside the patch are simplified. If the border is still preserved, the reconstruction in 3D will not suffer from artifacts. The high-resolution geometry image can be used to reverse the simplification and improve the connectivity at the decoder side. Three embodiments of patch mesh simplification are described: quadric error edge collapse, border distance edge collapse, and border triangles only.

Abstract: A new high-level syntax element referred to as a basemesh patch data unit allows the transmission of syntax elements related to the basemesh component of a V3C bitstream. Previously, the V3C standard did not have any additional patches targeting the basemesh only. A basemesh patch data unit works with the previously disclosed sub-patch concept to provide syntax elements to be used with basemesh bitstream. The basemesh patch data unit allows mesh data and other types of data (e.g., point clouds) to be mixed together at the patch level and generate a richer representation of 3D objects. Furthermore, the basemesh patch data unit expands the concept of sub-meshes and patches by providing a flexible way to arrange the data in 3D (at the basemesh level) and in 2D (at the texture map level) allowing different configurations, instance, multiple attribute images and multiple basemeshes.

Abstract: An attribute prediction and compensation scheme for geometry-based dynamic point cloud compression is described herein. A combination of multiple reference frames are able to be used as a predictor for current frames. The method described herein improves efficiency and accuracy.

Abstract: An electronic device and method for a variable rate compression of a point cloud geometry is provided. The electronic device stores a set of RD operation points and coding modes associated with the set of RD operation points. The electronic device receives a 3D point cloud geometry and partitions the 3D geometry into a set of blocks. After the partition, the electronic device selects a block and computes a set of loss values associated with one or more compression metrics. Such loss values correspond to a set of coding modes associated with at least a subset of the set of RD operation points. From the set of coding modes, the electronic device selects a coding mode for which a loss value of the set of loss values is below a loss threshold for that coding mode. Thereafter, the electronic device encodes the block based on the coding mode.

Abstract: The single Level-of-Detail (LoD) per block displacement packing method enables slice decoding, scalability and other video processing implementations. The sequence displacement bitrate is only minimally changed by using the LoD per block packing implementation.

Abstract: With the concept of sub-patches, the single geometry is described at the patch level, while the multiple attributes are then described at the sub-patch level. One application of the concept is the derivation of texture coordinates using projections for several sections of a mesh surface (affecting the attributes only), whereby the geometry of the whole mesh section uses just one single syntax element.

Abstract: An electronic device and method for detection and indication of geometry reconstruction artifacts is provided. The electronic device acquires a reference point cloud, encodes the reference point cloud to generate encoded point cloud data, and decodes the encoded point cloud data to generate a test point cloud. The electronic device further generates a first local density map and a second local density map for points of the reference point cloud and the test point cloud, respectively. The electronic device generates a final density map based on the first local density map and the second local density map, and further generates supplementary information based on the final density map. The supplementary information includes missing points data corresponding to regions of the test point cloud that include artifacts such as holes or includes descriptors for the regions that include the artifacts. The electronic device signals the supplementary information to a decoder.

Abstract: A new SEI message for the V-DMC standard is described herein, the zippering SEI. The zippering SEI message can be used by the decoder for the mesh reconstruction, where in the case of multiple sub-meshes, the zippering SEI provides ways to reduce common artifacts caused by independent sub-mesh encoding, such as holes and cracks on the mesh surface.

Abstract: In the current implementation of V-DMC, the (u, v) coordinates are generated using Microsoft UVAtlas and they, together with the 3D positions and the topology, are carried in the base mesh sub-bitstream. High level syntax structures described herein support projection-based atlas map generation, and the means to derive the (u, v) coordinates on the decoder side using V3C syntax structure extensions. In comparison with previous implementations and in order to preserve the current V3C geometry bitstream concept, a separate sub-bitstream referred to hereby as vertex property sub-bitstream is used to carry displacement information.

Abstract: The generation of a texture map using orthographic projections is performed in a fast and efficient manner. A method to generate texture maps taking significantly less time and also allowing maps to exploit the correlation between content of different frames in time is described herein. The texture mapping is able to be used for automatic generation of volumetric content or for more efficient compression of dynamic meshes. The texture map generation described herein includes ways to generate a texture atlas using orthographic projections. A novel stretch metric for orthographic projections is described, and a merging algorithm is devised to optimally cluster triangles into a single patch. Additionally, packing techniques are able to be used for mesh patches that try to optimize size and temporal stability.

Abstract: A method is disclosed to generate (u,v) coordinates at the decoder side by using parameters of orthographic projection functions, transmitted via an atlas bitstream. With the parameters for orthographic projection, the decoder is able to efficiently generate (u,v) coordinates and avoid their expensive coding.

Abstract: A method of compressing meshes using a projection-based approach, leveraging and expanding the tools and syntax generated for projection-based volumetric content compression is described. The mesh is segmented into surface patches, with the difference that the segments follow the connectivity of the mesh. The dense mesh compression utilizes 3D surface patches to represent connected triangles on a mesh surface and groups of vertices to represent triangles not captured by surface projection. Each surface patch (or 3D patch) is projected to a 2D patch, whereby for the mesh, the triangle surface sampling is similar to a common rasterization approach. For each patch, position and connectivity of the projected vertices are kept. The sampled surface resembles a point cloud and is coded with the same approach used for point cloud compression. The list of vertices and connectivity per patch is encoded, and the data is sent with the coded point cloud data.

Abstract: A method of compression of 3D mesh data using projections of mesh surface data and video representation of connectivity data is described herein. The method utilizes 3D surface patches to represent a set of connected triangles on a mesh surface. The projected surface data is stored in patches (a mesh patch) that is encoded in atlas data. The connectivity of the mesh, that is, the vertices and the triangles of the surface patch, are encoded using video-based compression techniques. The data is encapsulated in a new video component named vertex video data, and the disclosed structure allows for progressive mesh coding by separating sets of vertices in layers, and creating levels of detail for the mesh connectivity. This approach extends the functionality of the V3C (volumetric video-based) standard, currently being used for coding of point cloud and multiview plus depth content.

Abstract: The connectivity information and mapping information of a mesh surface patch are able to be encoded after projection to 2D. Regarding the connectivity information, the projection operation does not change the connection between vertices, so the same list of connected vertices are able to be carried in the atlas data. Similarly, the mapping information does not change after projection and is able to be carried in the atlas data. Two methods are disclosed for encoding the connectivity and mapping information. For the connectivity information, a video-based method uses neighboring color coding. For mapping coordinates, a method uses the projected vertex position. The connectivity and mapping are also able to be processed by an external mesh encoder. Newly proposed mapping information is able to be taken advantage of to perform temporal compression.

Abstract: Ways to post-process a decoded mesh and modify received triangles per patch to improve the mesh geometry are described herein. Since the transmitted geometry contains the high-resolution surface information, the information is able to be applied to the mesh reconstruction operation to generate triangles that are aligned with the original surface. Methods include generating new triangles by splitting the received triangles' edges according to their size, by inserting new vertices at the triangle's centroids, by splitting the vertices, and by performing marching cubes in surfaces defined by the geometry images.

Abstract: Depth image generation is improved by more efficient encoding using video codecs. The mapping of the depth to the luma channel is performed by not using all bits available, and with the remaining bits, a depth scaling factor is generated and incorporated into the bilinear interpolation algorithm used during rasterization. A normal filtering procedure is described, where the positions of vertices are adjusted according to the normal estimated from the surface pixels. After decoding the depth image, the pixels related to the surface of a triangle are collected and used to estimate a plane and the normal of the plane. The normal is compared to the normal obtained from the plane defined by the three vertices of the triangle. If there is no match, the positions of the vertices are adjusted to match the estimated normal from the pixels' surfaces. The adjustment can follow an iterative minimization process.