Patents by Inventor Yifu Zhang
Yifu Zhang has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20200342677Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.Type: ApplicationFiled: July 9, 2020Publication date: October 29, 2020Applicant: Magic Leap, Inc.Inventors: David Geoffrey MOLYNEAUX, Frank Thomas STEINBRÜCKER, Zhongle WU, Xiaolin WEI, Jianyuan MIN, Yifu ZHANG
-
Patent number: 10713852Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.Type: GrantFiled: December 21, 2018Date of Patent: July 14, 2020Assignee: Magic Leap, Inc.Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrucker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
-
Publication number: 20200202633Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.Type: ApplicationFiled: March 4, 2020Publication date: June 25, 2020Applicant: Magic Leap, Inc.Inventors: Frank Thomas Steinbrücker, David Geoffrey Molyneaux, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
-
Patent number: 10636219Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.Type: GrantFiled: December 21, 2018Date of Patent: April 28, 2020Assignee: Magic Leap, Inc.Inventors: Frank Thomas Steinbrücker, David Geoffrey Molyneaux, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
-
Publication number: 20200126298Abstract: A method for forming a reconstructed 3D mesh includes receiving a set of captured depth maps associated with a scene, performing an initial camera pose alignment associated with the set of captured depth maps, and overlaying the set of captured depth maps in a reference frame. The method also includes detecting one or more shapes in the overlaid set of captured depth maps and updating the initial camera pose alignment to provide a shape-aware camera pose alignment. The method further includes performing shape-aware volumetric fusion and forming the reconstructed 3D mesh associated with the scene.Type: ApplicationFiled: December 18, 2019Publication date: April 23, 2020Applicant: Magic Leap, Inc.Inventors: Xiaolin Wei, Yifu Zhang
-
Patent number: 10559127Abstract: A method for forming a reconstructed 3D mesh includes receiving a set of captured depth maps associated with a scene, performing an initial camera pose alignment associated with the set of captured depth maps, and overlaying the set of captured depth maps in a reference frame. The method also includes detecting one or more shapes in the overlaid set of captured depth maps and updating the initial camera pose alignment to provide a shape-aware camera pose alignment. The method further includes performing shape-aware volumetric fusion and forming the reconstructed 3D mesh associated with the scene.Type: GrantFiled: September 23, 2016Date of Patent: February 11, 2020Assignee: Magic Leap, Inc.Inventors: Xiaolin Wei, Yifu Zhang
-
Patent number: 10523933Abstract: Control data for a motion-constrained tile set (“MCTS”) indicates that inter-picture prediction processes within a specified set of tiles are constrained to reference only regions within the same set of tiles in previous pictures in decoding (or encoding) order. For example, a video encoder encodes multiple pictures partitioned into tiles to produce encoded data. The encoder outputs the encoded data along with control data (e.g., in a supplemental enhancement information message) that indicates that inter-picture prediction dependencies across tile set boundaries are constrained for a given tile set of one or more of the tiles. A video decoder or other tool receives the encoded data and MCTS control data, and processes the encoded data. Signaling and use of MCTS control data can facilitate region-of-interest decoding and display, transcoding to limit encoded data to a selected set of tiles, loss robustness, parallelism in encoding and/or decoding, and other video processing.Type: GrantFiled: July 13, 2017Date of Patent: December 31, 2019Assignee: Microsoft Technology Licensing, LLCInventors: Yongjun Wu, Gary J. Sullivan, Yifu Zhang
-
Publication number: 20190197786Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.Type: ApplicationFiled: December 21, 2018Publication date: June 27, 2019Applicant: Magic Leap, Inc.Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrücker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
-
Publication number: 20190197777Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.Type: ApplicationFiled: December 21, 2018Publication date: June 27, 2019Applicant: Magic Leap, Inc.Inventors: Frank Thomas Steinbrücker, David Geoffrey Molyneaux, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
-
Publication number: 20190197765Abstract: An augmented reality/mixed reality system that provides a more immersive user experience. That experience is provided with increased speed of update for occlusion data by using depth sensor data augmented with lower-level reconstruction data. When operating in real-time dynamic environments, changes in the physical world can be reflected quickly in the occlusion data. Occlusion rendering using live depth data augmented with lower-level 3D reconstruction data, such as a raycast point cloud, can greatly reduce the latency for visual occlusion processing. Generating occlusion data in this way may provide faster operation of an XR system using less computing resources and enabling the system to be packaged in a battery operated wearable device.Type: ApplicationFiled: December 21, 2018Publication date: June 27, 2019Applicant: Magic Leap, Inc.Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrücker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
-
Publication number: 20190197774Abstract: A method of operating a computing system to generate a model of an environment represented by a mesh is provided. The method allows to update 3D meshes to client applications in real time with low latency to support on the fly environment changes. The method provides 3D meshes adaptive to different levels of simplification requested by various client applications. The method provides local update, for example, updating the mesh parts that are changed since last update. The method also provides 3D meshes with planarized surfaces to support robust physics simulations. The method includes segmenting a 3D mesh into mesh blocks. The method also includes performing a multi-stage simplification on selected mesh blocks. The multi-stage simplification includes a pre-simplification operation, a planarization operation, and a post-simplification operation.Type: ApplicationFiled: December 21, 2018Publication date: June 27, 2019Applicant: Magic Leap, Inc.Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrücker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
-
Patent number: 10249087Abstract: A method of atlas packing includes receiving a three-dimensional (3D) mesh that includes a plurality of triangles representing surfaces of one or more objects; for each respective triangle, determining a normal of the respective triangle, and categorizing the respective triangle into one of six directions along positive and negative of x-, y-, and z-directions; categorizing triangles in each respective direction into one or more layers orthogonal to the respective direction; for each respective layer, identifying one or more connected components; projecting each respective connected component onto a plane orthogonal to the respective direction to obtain a corresponding projected two-dimensional (2D) connected component; cutting the projected 2D connected component into one or more sub-components; packing the bounding boxes of all sub-components into one or more atlases; and for each respective triangle of each sub-component, copying a texture of a corresponding triangle of the 3D mesh to the respective triangType: GrantFiled: January 23, 2017Date of Patent: April 2, 2019Assignee: MAGIC LEAP, INC.Inventors: Xiaolin Wei, Yifu Zhang
-
Publication number: 20170318288Abstract: Control data for a motion-constrained tile set (“MCTS”) indicates that inter-picture prediction processes within a specified set of tiles are constrained to reference only regions within the same set of tiles in previous pictures in decoding (or encoding) order. For example, a video encoder encodes multiple pictures partitioned into tiles to produce encoded data. The encoder outputs the encoded data along with control data (e.g., in a supplemental enhancement information message) that indicates that inter-picture prediction dependencies across tile set boundaries are constrained for a given tile set of one or more of the tiles. A video decoder or other tool receives the encoded data and MCTS control data, and processes the encoded data. Signaling and use of MCTS control data can facilitate region-of-interest decoding and display, transcoding to limit encoded data to a selected set of tiles, loss robustness, parallelism in encoding and/or decoding, and other video processing.Type: ApplicationFiled: July 13, 2017Publication date: November 2, 2017Applicant: Microsoft Technology Licensing, LLCInventors: Yongjun Wu, Gary J. Sullivan, Yifu Zhang
-
Patent number: 9749627Abstract: Control data for a motion-constrained tile set (“MCTS”) indicates that inter-picture prediction processes within a specified set of tiles are constrained to reference only regions within the same set of tiles in previous pictures in decoding (or encoding) order. For example, a video encoder encodes multiple pictures partitioned into tiles to produce encoded data. The encoder outputs the encoded data along with control data (e.g., in a supplemental enhancement information message) that indicates that inter-picture prediction dependencies across tile set boundaries are constrained for a given tile set of one or more of the tiles. A video decoder or other tool receives the encoded data and MCTS control data, and processes the encoded data. Signaling and use of MCTS control data can facilitate region-of-interest decoding and display, transcoding to limit encoded data to a selected set of tiles, loss robustness, parallelism in encoding and/or decoding, and other video processing.Type: GrantFiled: September 9, 2013Date of Patent: August 29, 2017Assignee: Microsoft Technology Licensing, LLCInventors: Yongjun Wu, Gary J. Sullivan, Yifu Zhang
-
Publication number: 20170221263Abstract: A method of atlas packing includes receiving a three-dimensional (3D) mesh that includes a plurality of triangles representing surfaces of one or more objects; for each respective triangle, determining a normal of the respective triangle, and categorizing the respective triangle into one of six directions along positive and negative of x-, y-, and z-directions; categorizing triangles in each respective direction into one or more layers orthogonal to the respective direction; for each respective layer, identifying one or more connected components; projecting each respective connected component onto a plane orthogonal to the respective direction to obtain a corresponding projected two-dimensional (2D) connected component; cutting the projected 2D connected component into one or more sub-components; packing the bounding boxes of all sub-components into one or more atlases; and for each respective triangle of each sub-component, copying a texture of a corresponding triangle of the 3D mesh to the respective triangType: ApplicationFiled: January 23, 2017Publication date: August 3, 2017Applicant: Magic Leap, Inc.Inventors: Xiaolin Wei, Yifu Zhang
-
Patent number: 9674506Abstract: A system that incorporates teachings of the exemplary embodiments may include, for example, means for generating a disparity map based on a depth map, means for determining accuracy of pixels in the depth map where the determining means identifies the pixels as either accurate or inaccurate based on a confidence map and the disparity map, and means for providing an adjusted depth map where the providing means adjusts inaccurate pixels of the depth map using a cost function associated with the inaccurate pixels. Other embodiments are disclosed.Type: GrantFiled: September 3, 2015Date of Patent: June 6, 2017Assignee: AT&T Intellectual Property I, L.P.Inventors: Chao Tian, Vinay Anant Vaishampayan, Yifu Zhang
-
Publication number: 20170091996Abstract: A method for forming a reconstructed 3D mesh includes receiving a set of captured depth maps associated with a scene, performing an initial camera pose alignment associated with the set of captured depth maps, and overlaying the set of captured depth maps in a reference frame. The method also includes detecting one or more shapes in the overlaid set of captured depth maps and updating the initial camera pose alignment to provide a shape-aware camera pose alignment. The method further includes performing shape-aware volumetric fusion and forming the reconstructed 3D mesh associated with the scene.Type: ApplicationFiled: September 23, 2016Publication date: March 30, 2017Applicant: Magic Leap, Inc.Inventors: Xiaolin Wei, Yifu Zhang
-
Publication number: 20150381966Abstract: A system that incorporates teachings of the exemplary embodiments may include, for example, means for generating a disparity map based on a depth map, means for determining accuracy of pixels in the depth map where the determining means identifies the pixels as either accurate or inaccurate based on a confidence map and the disparity map, and means for providing an adjusted depth map where the providing means adjusts inaccurate pixels of the depth map using a cost function associated with the inaccurate pixels. Other embodiments are disclosed.Type: ApplicationFiled: September 3, 2015Publication date: December 31, 2015Inventors: CHAO TIAN, VINAY ANANT VAISHAMPAYAN, YIFU ZHANG
-
Patent number: 9153018Abstract: A system that incorporates teachings of the exemplary embodiments may include, for example, means for generating a disparity map based on a depth map, means for determining accuracy of pixels in the depth map where the determining means identifies the pixels as either accurate or inaccurate based on a confidence map and the disparity map, and means for providing an adjusted depth map where the providing means adjusts inaccurate pixels of the depth map using a cost function associated with the inaccurate pixels. Other embodiments are disclosed.Type: GrantFiled: January 30, 2015Date of Patent: October 6, 2015Assignee: AT&T Intellectual Property I, LPInventors: Chao Tian, Vinay Anant Vaishampayan, Yifu Zhang
-
Patent number: 9143776Abstract: Techniques for objectively determining perceived video/image quality, the techniques including receiving a degraded bit-stream comprising encoded video/image data, and subsequently parsing the bit-stream to extract one or more video/image coding components. The video coding components may include intra-prediction modes, discrete cosine transform (DCT) coefficients, motion information, or combinations thereof, and may be used as a basis for objectively predicting a Quality of Experience (QoE) or Motion Opinion Score (MOS) score of the degraded bit-stream.Type: GrantFiled: May 7, 2012Date of Patent: September 22, 2015Assignee: Futurewei Technologies, Inc.Inventors: Yifu Zhang, Dong-Qing Zhang, Hong Yu