Abstract: Methods, systems and devices described herein implement a task-driven machine learning-based compression scheme for point cloud geometry implicit representation. The machine learning-based codec is able to be optimized for a task to achieve better compression rates by being conditioned to what the reconstructed signal will be used for. The latent representation of the point cloud or the neural network that implicitly represents the point cloud itself are able to be compressed. The methods described herein perform efficient compression of the implicit representation of a point cloud given a target task.

Abstract: A novel method of classifying point cloud data by extending the Gray-level Co-occurrence Matrix (GLCM) technique from the 2D to the sparse 3D domain is described herein. The method is able to be applied to point clouds derived from a mesh collection/meshes (such as, the Real-World Textured Things (RWTT) mesh collection). Implementations designed for multiple purposes are described herein: sampling and quantization of RWTT meshes, generation of GLCMs and corresponding texture descriptors, and the selection of potential candidate point clouds based on these extracted descriptors.

Abstract: An electronic device and method for adaptive mode selection for point cloud compression, is provided. The electronic device receives a 3D point cloud geometry and partitions the 3D point cloud geometry into a set of 3D blocks. For a 3D block of the set of 3D blocks, mode decision information is determined. The mode decision information includes class information of the 3D point cloud geometry, operational conditions associated with an encoding stage of the 3D point cloud geometry, or mode-related information associated with one or more 3D blocks of the set of 3D blocks. Based on the mode decision information, one or more modes are selected for the 3D block from a plurality of modes. Each mode corresponds to a function that is used to encode the 3D block. The 3D block is encoded based on the one or more modes.

Abstract: Described herein is a method to segment meshes into sub-meshes based on triangle properties. The triangles are first classified according to some characteristic using their respective areas. A filtering process may change the classification according to the neighboring triangles. Then connected components are generated, and neighboring connected components are merged following a certain criteria. In some embodiments, connected components that share the most amount of edges are merged. With this technique, sub-meshes can be automatically generated without any previous knowledge of the mesh generation stage.

Abstract: Ways to simplify connectivity data for patches are described herein. The patches are generated considering the high-resolution mesh information. The connectivity data is simplified at the patch level, while the geometry image is still preserved. For the connectivity simplification, only triangles inside the patch are simplified. If the border is still preserved, the reconstruction in 3D will not suffer from artifacts. The high-resolution geometry image can be used to reverse the simplification and improve the connectivity at the decoder side. Three embodiments of patch mesh simplification are described: quadric error edge collapse, border distance edge collapse, and border triangles only.

Abstract: An attribute prediction and compensation scheme for geometry-based dynamic point cloud compression is described herein. A combination of multiple reference frames are able to be used as a predictor for current frames. The method described herein improves efficiency and accuracy.

Abstract: A new SEI message for the V-DMC standard is described herein, the zippering SEI. The zippering SEI message can be used by the decoder for the mesh reconstruction, where in the case of multiple sub-meshes, the zippering SEI provides ways to reduce common artifacts caused by independent sub-mesh encoding, such as holes and cracks on the mesh surface.

Abstract: An artificial intelligence (AI) based method for calibration of a gaming ecosystem and to maximize satisfaction of a player involved in a gaming session such as single-player gaming session, multi-player online gaming session, and a videogame recommendation system is provided. The system detects the gaming session that includes an execution of a video game for a first gameplay and acquires sensor data associated with a player involved in the first gameplay. The sensor data corresponds to a duration of the detected gaming session. The system determines one or more indicators of a dissatisfaction of the player with the first gameplay based on application of one or more Artificial Intelligence (AI) models on the sensor data. The system controls the execution of the video game to modify one or more aspects associated with the first gameplay or a second gameplay of the video game that is different from the first gameplay.

Abstract: A method of compression of 3D mesh data using projections of mesh surface data and video representation of connectivity data is described herein. The method utilizes 3D surface patches to represent a set of connected triangles on a mesh surface. The projected surface data is stored in patches (a mesh patch) that is encoded in atlas data. The connectivity of the mesh, that is, the vertices and the triangles of the surface patch, are encoded using video-based compression techniques. The data is encapsulated in a new video component named vertex video data, and the disclosed structure allows for progressive mesh coding by separating sets of vertices in layers, and creating levels of detail for the mesh connectivity. This approach extends the functionality of the V3C (volumetric video-based) standard, currently being used for coding of point cloud and multiview plus depth content.

Abstract: Ways to simplify connectivity data for patches are described herein. The patches are generated considering the high-resolution mesh information. The connectivity data is simplified at the patch level, while the geometry image is still preserved. For the connectivity simplification, only triangles inside the patch are simplified. If the border is still preserved, the reconstruction in 3D will not suffer from artifacts. The high-resolution geometry image can be used to reverse the simplification and improve the connectivity at the decoder side. Three embodiments of patch mesh simplification are described: quadric error edge collapse, border distance edge collapse, and border triangles only.

Abstract: New syntax elements are used to extend patch types, and the syntax is added to the V3C standard. The new syntax defines patches that encode meshes by projecting connected triangles to a 2D surface, patches that encode triangles or triangle strips without any projection, or patches that are tracked over time and are encoded by projecting connected triangles to a 2D surface. Furthermore, the syntax allows for different ways of coding the mesh-specific information. For instance, the syntax enables three different encoding methods for the vertex position: explicit (directly added to the atlas stream), embedded on video data (occupancy map data), or encoded using an external mesh encoder.

Abstract: The connectivity information and mapping information of a mesh surface patch are able to be encoded after projection to 2D. Regarding the connectivity information, the projection operation does not change the connection between vertices, so the same list of connected vertices are able to be carried in the atlas data. Similarly, the mapping information does not change after projection and is able to be carried in the atlas data. Two methods are disclosed for encoding the connectivity and mapping information. For the connectivity information, a video-based method uses neighboring color coding. For mapping coordinates, a method uses the projected vertex position. The connectivity and mapping are also able to be processed by an external mesh encoder. Newly proposed mapping information is able to be taken advantage of to perform temporal compression.

Abstract: Ways to post-process a decoded mesh and modify received triangles per patch to improve the mesh geometry are described herein. Since the transmitted geometry contains the high-resolution surface information, the information is able to be applied to the mesh reconstruction operation to generate triangles that are aligned with the original surface. Methods include generating new triangles by splitting the received triangles' edges according to their size, by inserting new vertices at the triangle's centroids, by splitting the vertices, and by performing marching cubes in surfaces defined by the geometry images.

Abstract: Ways to improve mesh reconstruction by modifying the position of vertices at the border of patches to make sure that neighboring patches do not have a gap between them, also known as zippering, are described herein. Six different methods to implement the post-processing operation, as well as syntax elements and semantics for transmission of the filter parameters, are disclosed. A hierarchical method indicate the geometry distortion that can generate gaps between patches. The value per frame, or per patch, or per boundary object is sent. The number of bits to encode the values is also dependent on the previous geometry distortion. A method sends index matches instead of geometry distortion. The matching index is sent per boundary vertex, but a method to send only one index of the pair is implemented as well.

Abstract: An architecture includes new blocks that transform mesh connectivity to enable lossy compression. In addition, a method generates surface patches from meshes and preserves the overall aspect of the object, while generating patches for efficient encoding using the V3C standard. Triangles are joined together into connected components, and conditional projection of each triangle to the surface is performed to enforce connectivity at the projected surface. A new depth filtering algorithm is able to be used to improve compression performance. The new blocks added for connectivity compression and the techniques introduced to ensure 2D connectivity to allow for UV texture map transmission and to filter depth values and avoid highfrequency edges in the depth image are described herein.

Abstract: Depth image generation is improved by more efficient encoding using video codecs. The mapping of the depth to the luma channel is performed by not using all bits available, and with the remaining bits, a depth scaling factor is generated and incorporated into the bilinear interpolation algorithm used during rasterization. A normal filtering procedure is described, where the positions of vertices are adjusted according to the normal estimated from the surface pixels. After decoding the depth image, the pixels related to the surface of a triangle are collected and used to estimate a plane and the normal of the plane. The normal is compared to the normal obtained from the plane defined by the three vertices of the triangle. If there is no match, the positions of the vertices are adjusted to match the estimated normal from the pixels' surfaces. The adjustment can follow an iterative minimization process.

Abstract: A method and Video-Based Point Cloud Compression (V-PCC) decoder for synchronization of decoded frames before point cloud reconstruction is provided. A V-PCC bit-stream which includes encoded frames associated with a point cloud sequence is received. Sub-streams of the received V-PCC bit-stream are decoded by a group of video decoders of the V-PCC decoder to generate V-PCC components, such as an attribute component, a geometry component, an occupancy map component, and an atlas component. A release of the attribute component, the geometry component, the occupancy map component, and the atlas component to the reconstruction unit is delayed based on a first output delay, a second output delay, a third output delay, and a fourth output delay, respectively. The delayed release synchronizes the attribute component, the geometry component, the occupancy map component, and the atlas component with each other before the reconstruction unit reconstructs a point cloud based on the V-PCC components.

Abstract: Methods, systems and devices described herein implement a task-driven machine learning-based compression scheme for point cloud geometry implicit representation. The machine learning-based codec is able to be optimized for a task to achieve better compression rates by being conditioned to what the reconstructed signal will be used for. The latent representation of the point cloud or the neural network that implicitly represents the point cloud itself are able to be compressed. The methods described herein perform efficient compression of the implicit representation of a point cloud given a target task.

Abstract: Occupancy networks enable efficient and flexible point cloud compression. In addition to the voxel-based representation, occupancy networks are able to handle points, meshes, or projected images of 3D objects, making them very flexible in terms of input signal representation. The probability of occupancy of positions is estimated using occupancy networks instead of sparse convolutional neural networks. A compression implementation using occupancy network enables scalability with infinite reconstruction resolution.

Abstract: Methods, systems and device for efficiently compressing task-oriented dynamic meshes using occupancy networks are described herein. A single trained occupancy network model is able to reconstruct a mesh video using a few additional points per input mesh frame. To optimize the compression of the model and points, the estimated rate to compress the occupancy network is able to be included in the loss function. This minimizes the number of bits to encode the model, while it tries to reproduce the meshes as well as possible. An adaptive subsampling per input mesh is added to optimize the mesh reconstruction and the N-point point clouds compression. To optimize the model to perform a particular task, a metric is added to the cost function that takes this task into account.