Abstract: An electronic device and method for using a rate adaptive codec for dynamic point cloud compression is provided. The electronic device receives a point cloud frame (PCF) sequence that includes a set of reference PCFs and a current PCF that is to be encoded. The electronic device generates 3D feature data based on application of a neural network-based encoder on at least one of the set of reference PCFs and the current PCF. The electronic device selects from an encoding pipeline, an encoding path as an I-frame encoding path or a P-frame encoding path for the current PCF. The selected encoding path extends from an output of the neural network-based encoder and includes a first sequence of transformation operations that is applicable on the 3D feature data. The electronic device generates a bit-stream of coded PCF based on application of the first sequence of transformation operations on the 3D feature data.

Abstract: Post training compression of 3DGS data is agnostic to training in a traditional signal compression perspective. Gaussian parameters are treated as signals. Pre-processing and transform coding techniques are used to compress the signals effectively. Firstly, lossless/lossy compression is performed on 3DGS geometry (positions, scales, rotations) using a point cloud coding-based (e.g., G-PCC, GeS) framework. Positions are compressed using occupancy tree coding. Scales and rotations are encoded as attributes using transform coding. The widely used block-based graph Fourier transform (GFT) is used to compress the attributes (base colors, spherical harmonic coefficients and opacities). In addition, a graph construction strategy is used for 3DGS data that computes the edge weights based on similarity (or dissimilarity) between the 3D Gaussian distributions using KL-divergence. Alternatively, positions can be encoded using occupancy tree (e.g.

Abstract: Ways to improve mesh reconstruction by modifying the position of vertices at the border of patches to make sure that neighboring patches do not have a gap between them, also known as zippering, are described herein. Six different methods to implement the post-processing operation, as well as syntax elements and semantics for transmission of the filter parameters, are disclosed. A hierarchical method indicate the geometry distortion that can generate gaps between patches. The value per frame, or per patch, or per boundary object is sent. The number of bits to encode the values is also dependent on the previous geometry distortion. A method sends index matches instead of geometry distortion. The matching index is sent per boundary vertex, but a method to send only one index of the pair is implemented as well.

Abstract: Techniques to handle the Spherical Harmonic (SH) coefficients associated with 3DGS towards effective compression are described herein. SH coefficients are used to represent view dependent RGB color values, which takes a significant amount of memory—48 out of 59 parameters per Gaussian. By applying suitable transformation to SH coefficients, the inter-channel redundancies are exploited, thereby effectively reducing the memory requirements by 50%.

Abstract: An attribute inter-prediction framework using motion search criterion, proper reference, attribute projection and in-loop filter is described herein. An alternative motion estimation strategy incorporates color distance into the motion search criterion where geometry and attribute motion are jointly searched which leads to a single motion vector associated with each predicted block. The composed distance is the weighted sum of color and geometry distance. With the composed distance, two new types of search criterion are used to estimate the motion vectors: using the nearest neighbor composed distance, using the minimum composed distance based on the search of composed distances associated with each reference point within the search window. A dedicated attribute coding scheme using a two-pass motion search scheme, where the first pass is based on geometry only, and the second pass targets on eliminating temporal attribute redundancy. An in-loop filter can be applied to the projected attributes.

Abstract: The connectivity information and mapping information of a mesh surface patch are able to be encoded after projection to 2D. Regarding the connectivity information, the projection operation does not change the connection between vertices, so the same list of connected vertices are able to be carried in the atlas data. Similarly, the mapping information does not change after projection and is able to be carried in the atlas data. Two methods are disclosed for encoding the connectivity and mapping information. For the connectivity information, a video-based method uses neighboring color coding. For mapping coordinates, a method uses the projected vertex position. The connectivity and mapping are also able to be processed by an external mesh encoder. Newly proposed mapping information is able to be taken advantage of to perform temporal compression.

Abstract: Generating and sharing interactive three dimensional (3D) content/3D models captured from gaming sessions are described herein. The 3D content is able to be shared with mobile devices, televisions, gaming consoles, Virtual Reality (VR) devices or other devices. Since the content is rendered in 3D, a user is able to change the view direction, zoom in/out, and perform other functions. A framework enables video game media galleries to capture, view, edit, and share interactive, static or dynamic 3D media while keeping the structure of the original gaming assets inaccessible to the end-user.

Abstract: Improvements to orthoAtlas include: new syntax elements to indicate the mapping method used by orthoAtlas; a method to adjust projection parameters to compensate for compression artifacts in encoded vertex positions; a syntax modification to efficiently encode projection parameters taking advantage of temporal correlation; a method to reduce signaling by using the calculated bounding box to derive projection parameters; and a method to avoid bounding box estimation at decoder side, therefore reducing the decoder complexity.

Abstract: Depth image generation is improved by more efficient encoding using video codecs. The mapping of the depth to the luma channel is performed by not using all bits available, and with the remaining bits, a depth scaling factor is generated and incorporated into the bilinear interpolation algorithm used during rasterization. A normal filtering procedure is described, where the positions of vertices are adjusted according to the normal estimated from the surface pixels. After decoding the depth image, the pixels related to the surface of a triangle are collected and used to estimate a plane and the normal of the plane. The normal is compared to the normal obtained from the plane defined by the three vertices of the triangle. If there is no match, the positions of the vertices are adjusted to match the estimated normal from the pixels' surfaces. The adjustment can follow an iterative minimization process.

Abstract: New syntax elements are used to extend patch types, and the syntax is added to the V3C standard. The new syntax defines patches that encode meshes by projecting connected triangles to a 2D surface, patches that encode triangles or triangle strips without any projection, or patches that are tracked over time and are encoded by projecting connected triangles to a 2D surface. Furthermore, the syntax allows for different ways of coding the mesh-specific information. For instance, the syntax enables three different encoding methods for the vertex position: explicit (directly added to the atlas stream), embedded on video data (occupancy map data), or encoded using an external mesh encoder.

Abstract: An attribute prediction and compensation scheme for geometry-based dynamic point cloud compression is described herein. A combination of multiple reference frames are able to be used as a predictor for current frames. The method described herein improves efficiency and accuracy.

Abstract: Methods, systems and device for efficiently compressing task-oriented dynamic meshes using occupancy networks are described herein. A single trained occupancy network model is able to reconstruct a mesh video using a few additional points per input mesh frame. To optimize the compression of the model and points, the estimated rate to compress the occupancy network is able to be included in the loss function. This minimizes the number of bits to encode the model, while it tries to reproduce the meshes as well as possible. An adaptive subsampling per input mesh is added to optimize the mesh reconstruction and the N-point point clouds compression. To optimize the model to perform a particular task, a metric is added to the cost function that takes this task into account.

Abstract: An electronic device and method for multiscale inter-prediction for dynamic point cloud compression is provided. The electronic device receives a set of reference point cloud frames and a current point cloud frame. The electronic device generates reference frame data comprising a feature set for each reference point cloud frame and a first set of features for the current point cloud frame. The electronic device predicts a second set of features for the current point cloud frame, using a first neural network predictor, based on the reference frame data. The electronic device computes a set of residual features based on the first set of features and the second set of features. The electronic device generates a set of quantized residual features based on the set of residual features and a bitstream of encoded point cloud data for the current 3D point cloud frame based on the set of quantized residual features.

Abstract: An architecture includes new blocks that transform mesh connectivity to enable lossy compression. In addition, a method generates surface patches from meshes and preserves the overall aspect of the object, while generating patches for efficient encoding using the V3C standard. Triangles are joined together into connected components, and conditional projection of each triangle to the surface is performed to enforce connectivity at the projected surface. A new depth filtering algorithm is able to be used to improve compression performance. The new blocks added for connectivity compression and the techniques introduced to ensure 2D connectivity to allow for UV texture map transmission and to filter depth values and avoid high-frequency edges in the depth image are described herein.

Abstract: Ways to post-process a decoded mesh and modify received triangles per patch to improve the mesh geometry are described herein. Since the transmitted geometry contains the high-resolution surface information, the information is able to be applied to the mesh reconstruction operation to generate triangles that are aligned with the original surface. Methods include generating new triangles by splitting the received triangles' edges according to their size, by inserting new vertices at the triangle's centroids, by splitting the vertices, and by performing marching cubes in surfaces defined by the geometry images.

Abstract: Methods, systems and devices described herein implement a task-driven machine learning-based compression scheme for point cloud geometry implicit representation. The machine learning-based codec is able to be optimized for a task to achieve better compression rates by being conditioned to what the reconstructed signal will be used for. The latent representation of the point cloud or the neural network that implicitly represents the point cloud itself are able to be compressed. The methods described herein perform efficient compression of the implicit representation of a point cloud given a target task.

Abstract: A novel method of classifying point cloud data by extending the Gray-level Co-occurrence Matrix (GLCM) technique from the 2D to the sparse 3D domain is described herein. The method is able to be applied to point clouds derived from a mesh collection/meshes (such as, the Real-World Textured Things (RWTT) mesh collection). Implementations designed for multiple purposes are described herein: sampling and quantization of RWTT meshes, generation of GLCMs and corresponding texture descriptors, and the selection of potential candidate point clouds based on these extracted descriptors.

Abstract: An electronic device and method for adaptive mode selection for point cloud compression, is provided. The electronic device receives a 3D point cloud geometry and partitions the 3D point cloud geometry into a set of 3D blocks. For a 3D block of the set of 3D blocks, mode decision information is determined. The mode decision information includes class information of the 3D point cloud geometry, operational conditions associated with an encoding stage of the 3D point cloud geometry, or mode-related information associated with one or more 3D blocks of the set of 3D blocks. Based on the mode decision information, one or more modes are selected for the 3D block from a plurality of modes. Each mode corresponds to a function that is used to encode the 3D block. The 3D block is encoded based on the one or more modes.

Abstract: Described herein is a method to segment meshes into sub-meshes based on triangle properties. The triangles are first classified according to some characteristic using their respective areas. A filtering process may change the classification according to the neighboring triangles. Then connected components are generated, and neighboring connected components are merged following a certain criteria. In some embodiments, connected components that share the most amount of edges are merged. With this technique, sub-meshes can be automatically generated without any previous knowledge of the mesh generation stage.

Abstract: Ways to simplify connectivity data for patches are described herein. The patches are generated considering the high-resolution mesh information. The connectivity data is simplified at the patch level, while the geometry image is still preserved. For the connectivity simplification, only triangles inside the patch are simplified. If the border is still preserved, the reconstruction in 3D will not suffer from artifacts. The high-resolution geometry image can be used to reverse the simplification and improve the connectivity at the decoder side. Three embodiments of patch mesh simplification are described: quadric error edge collapse, border distance edge collapse, and border triangles only.