Patents by Inventor Yifu Zhang

Yifu Zhang has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20210068646
    Abstract: An otoscope configured to be used together with a host device, such as a smart phone or other handheld mobile devices. The otoscope can utilize features on the hand-held mobile device, such as camera, software, and display, to allow a user view a person's ear canal and tympanic membrane or eardrum. In some embodiments, the otoscope may comprise a housing; a light source; an speculum; an optical redirection component; one or more image sensors; image-processing circuitry; and communication circuitry configured to transmit the sensed image to the host device.
    Type: Application
    Filed: April 2, 2019
    Publication date: March 11, 2021
    Inventors: Jane Yuqian ZHANG, Yuk Kwan LEE, Yifu ZHANG
  • Patent number: 10937246
    Abstract: A method of operating a computing system to generate a model of an environment represented by a mesh is provided. The method allows to update 3D meshes to client applications in real time with low latency to support on the fly environment changes. The method provides 3D meshes adaptive to different levels of simplification requested by various client applications. The method provides local update, for example, updating the mesh parts that are changed since last update. The method also provides 3D meshes with planarized surfaces to support robust physics simulations. The method includes segmenting a 3D mesh into mesh blocks. The method also includes performing a multi-stage simplification on selected mesh blocks. The multi-stage simplification includes a pre-simplification operation, a planarization operation, and a post-simplification operation.
    Type: Grant
    Filed: December 21, 2018
    Date of Patent: March 2, 2021
    Assignee: Magic Leap, Inc.
    Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrücker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Publication number: 20210056763
    Abstract: A method of operating a computing system to generate a model of an environment represented by a mesh is provided. The method allows to update 3D meshes to client applications in real time with low latency to support on the fly environment changes. The method provides 3D meshes adaptive to different levels of simplification requested by various client applications. The method provides local update, for example, updating the mesh parts that are changed since last update. The method also provides 3D meshes with planarized surfaces to support robust physics simulations. The method includes segmenting a 3D mesh into mesh blocks. The method also includes performing a multi-stage simplification on selected mesh blocks. The multi-stage simplification includes a pre-simplification operation, a planarization operation, and a post-simplification operation.
    Type: Application
    Filed: October 26, 2020
    Publication date: February 25, 2021
    Applicant: Magic Leap, Inc.
    Inventors: David Geoffrey MOLYNEAUX, Frank Thomas STEINBRÜCKER, Zhongle WU, Xiaolin WEI, Jianyuan MIN, Yifu ZHANG
  • Patent number: 10902679
    Abstract: An augmented reality/mixed reality system that provides a more immersive user experience. That experience is provided with increased speed of update for occlusion data by using depth sensor data augmented with lower-level reconstruction data. When operating in real-time dynamic environments, changes in the physical world can be reflected quickly in the occlusion data. Occlusion rendering using live depth data augmented with lower-level 3D reconstruction data, such as a raycast point cloud, can greatly reduce the latency for visual occlusion processing. Generating occlusion data in this way may provide faster operation of an XR system using less computing resources and enabling the system to be packaged in a battery operated wearable device.
    Type: Grant
    Filed: December 21, 2018
    Date of Patent: January 26, 2021
    Assignee: MAGIC LEAP, INC.
    Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrücker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Publication number: 20200342677
    Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.
    Type: Application
    Filed: July 9, 2020
    Publication date: October 29, 2020
    Applicant: Magic Leap, Inc.
    Inventors: David Geoffrey MOLYNEAUX, Frank Thomas STEINBRÜCKER, Zhongle WU, Xiaolin WEI, Jianyuan MIN, Yifu ZHANG
  • Patent number: 10713852
    Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.
    Type: Grant
    Filed: December 21, 2018
    Date of Patent: July 14, 2020
    Assignee: Magic Leap, Inc.
    Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrucker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Publication number: 20200202633
    Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.
    Type: Application
    Filed: March 4, 2020
    Publication date: June 25, 2020
    Applicant: Magic Leap, Inc.
    Inventors: Frank Thomas Steinbrücker, David Geoffrey Molyneaux, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Patent number: 10636219
    Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.
    Type: Grant
    Filed: December 21, 2018
    Date of Patent: April 28, 2020
    Assignee: Magic Leap, Inc.
    Inventors: Frank Thomas Steinbrücker, David Geoffrey Molyneaux, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Publication number: 20200126298
    Abstract: A method for forming a reconstructed 3D mesh includes receiving a set of captured depth maps associated with a scene, performing an initial camera pose alignment associated with the set of captured depth maps, and overlaying the set of captured depth maps in a reference frame. The method also includes detecting one or more shapes in the overlaid set of captured depth maps and updating the initial camera pose alignment to provide a shape-aware camera pose alignment. The method further includes performing shape-aware volumetric fusion and forming the reconstructed 3D mesh associated with the scene.
    Type: Application
    Filed: December 18, 2019
    Publication date: April 23, 2020
    Applicant: Magic Leap, Inc.
    Inventors: Xiaolin Wei, Yifu Zhang
  • Patent number: 10559127
    Abstract: A method for forming a reconstructed 3D mesh includes receiving a set of captured depth maps associated with a scene, performing an initial camera pose alignment associated with the set of captured depth maps, and overlaying the set of captured depth maps in a reference frame. The method also includes detecting one or more shapes in the overlaid set of captured depth maps and updating the initial camera pose alignment to provide a shape-aware camera pose alignment. The method further includes performing shape-aware volumetric fusion and forming the reconstructed 3D mesh associated with the scene.
    Type: Grant
    Filed: September 23, 2016
    Date of Patent: February 11, 2020
    Assignee: Magic Leap, Inc.
    Inventors: Xiaolin Wei, Yifu Zhang
  • Patent number: 10523933
    Abstract: Control data for a motion-constrained tile set (“MCTS”) indicates that inter-picture prediction processes within a specified set of tiles are constrained to reference only regions within the same set of tiles in previous pictures in decoding (or encoding) order. For example, a video encoder encodes multiple pictures partitioned into tiles to produce encoded data. The encoder outputs the encoded data along with control data (e.g., in a supplemental enhancement information message) that indicates that inter-picture prediction dependencies across tile set boundaries are constrained for a given tile set of one or more of the tiles. A video decoder or other tool receives the encoded data and MCTS control data, and processes the encoded data. Signaling and use of MCTS control data can facilitate region-of-interest decoding and display, transcoding to limit encoded data to a selected set of tiles, loss robustness, parallelism in encoding and/or decoding, and other video processing.
    Type: Grant
    Filed: July 13, 2017
    Date of Patent: December 31, 2019
    Assignee: Microsoft Technology Licensing, LLC
    Inventors: Yongjun Wu, Gary J. Sullivan, Yifu Zhang
  • Publication number: 20190197777
    Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.
    Type: Application
    Filed: December 21, 2018
    Publication date: June 27, 2019
    Applicant: Magic Leap, Inc.
    Inventors: Frank Thomas Steinbrücker, David Geoffrey Molyneaux, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Publication number: 20190197765
    Abstract: An augmented reality/mixed reality system that provides a more immersive user experience. That experience is provided with increased speed of update for occlusion data by using depth sensor data augmented with lower-level reconstruction data. When operating in real-time dynamic environments, changes in the physical world can be reflected quickly in the occlusion data. Occlusion rendering using live depth data augmented with lower-level 3D reconstruction data, such as a raycast point cloud, can greatly reduce the latency for visual occlusion processing. Generating occlusion data in this way may provide faster operation of an XR system using less computing resources and enabling the system to be packaged in a battery operated wearable device.
    Type: Application
    Filed: December 21, 2018
    Publication date: June 27, 2019
    Applicant: Magic Leap, Inc.
    Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrücker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Publication number: 20190197786
    Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.
    Type: Application
    Filed: December 21, 2018
    Publication date: June 27, 2019
    Applicant: Magic Leap, Inc.
    Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrücker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Publication number: 20190197774
    Abstract: A method of operating a computing system to generate a model of an environment represented by a mesh is provided. The method allows to update 3D meshes to client applications in real time with low latency to support on the fly environment changes. The method provides 3D meshes adaptive to different levels of simplification requested by various client applications. The method provides local update, for example, updating the mesh parts that are changed since last update. The method also provides 3D meshes with planarized surfaces to support robust physics simulations. The method includes segmenting a 3D mesh into mesh blocks. The method also includes performing a multi-stage simplification on selected mesh blocks. The multi-stage simplification includes a pre-simplification operation, a planarization operation, and a post-simplification operation.
    Type: Application
    Filed: December 21, 2018
    Publication date: June 27, 2019
    Applicant: Magic Leap, Inc.
    Inventors: David Geoffrey Molyneaux, Frank Thomas Steinbrücker, Zhongle Wu, Xiaolin Wei, Jianyuan Min, Yifu Zhang
  • Patent number: 10249087
    Abstract: A method of atlas packing includes receiving a three-dimensional (3D) mesh that includes a plurality of triangles representing surfaces of one or more objects; for each respective triangle, determining a normal of the respective triangle, and categorizing the respective triangle into one of six directions along positive and negative of x-, y-, and z-directions; categorizing triangles in each respective direction into one or more layers orthogonal to the respective direction; for each respective layer, identifying one or more connected components; projecting each respective connected component onto a plane orthogonal to the respective direction to obtain a corresponding projected two-dimensional (2D) connected component; cutting the projected 2D connected component into one or more sub-components; packing the bounding boxes of all sub-components into one or more atlases; and for each respective triangle of each sub-component, copying a texture of a corresponding triangle of the 3D mesh to the respective triang
    Type: Grant
    Filed: January 23, 2017
    Date of Patent: April 2, 2019
    Assignee: MAGIC LEAP, INC.
    Inventors: Xiaolin Wei, Yifu Zhang
  • Publication number: 20170318288
    Abstract: Control data for a motion-constrained tile set (“MCTS”) indicates that inter-picture prediction processes within a specified set of tiles are constrained to reference only regions within the same set of tiles in previous pictures in decoding (or encoding) order. For example, a video encoder encodes multiple pictures partitioned into tiles to produce encoded data. The encoder outputs the encoded data along with control data (e.g., in a supplemental enhancement information message) that indicates that inter-picture prediction dependencies across tile set boundaries are constrained for a given tile set of one or more of the tiles. A video decoder or other tool receives the encoded data and MCTS control data, and processes the encoded data. Signaling and use of MCTS control data can facilitate region-of-interest decoding and display, transcoding to limit encoded data to a selected set of tiles, loss robustness, parallelism in encoding and/or decoding, and other video processing.
    Type: Application
    Filed: July 13, 2017
    Publication date: November 2, 2017
    Applicant: Microsoft Technology Licensing, LLC
    Inventors: Yongjun Wu, Gary J. Sullivan, Yifu Zhang
  • Patent number: 9749627
    Abstract: Control data for a motion-constrained tile set (“MCTS”) indicates that inter-picture prediction processes within a specified set of tiles are constrained to reference only regions within the same set of tiles in previous pictures in decoding (or encoding) order. For example, a video encoder encodes multiple pictures partitioned into tiles to produce encoded data. The encoder outputs the encoded data along with control data (e.g., in a supplemental enhancement information message) that indicates that inter-picture prediction dependencies across tile set boundaries are constrained for a given tile set of one or more of the tiles. A video decoder or other tool receives the encoded data and MCTS control data, and processes the encoded data. Signaling and use of MCTS control data can facilitate region-of-interest decoding and display, transcoding to limit encoded data to a selected set of tiles, loss robustness, parallelism in encoding and/or decoding, and other video processing.
    Type: Grant
    Filed: September 9, 2013
    Date of Patent: August 29, 2017
    Assignee: Microsoft Technology Licensing, LLC
    Inventors: Yongjun Wu, Gary J. Sullivan, Yifu Zhang
  • Publication number: 20170221263
    Abstract: A method of atlas packing includes receiving a three-dimensional (3D) mesh that includes a plurality of triangles representing surfaces of one or more objects; for each respective triangle, determining a normal of the respective triangle, and categorizing the respective triangle into one of six directions along positive and negative of x-, y-, and z-directions; categorizing triangles in each respective direction into one or more layers orthogonal to the respective direction; for each respective layer, identifying one or more connected components; projecting each respective connected component onto a plane orthogonal to the respective direction to obtain a corresponding projected two-dimensional (2D) connected component; cutting the projected 2D connected component into one or more sub-components; packing the bounding boxes of all sub-components into one or more atlases; and for each respective triangle of each sub-component, copying a texture of a corresponding triangle of the 3D mesh to the respective triang
    Type: Application
    Filed: January 23, 2017
    Publication date: August 3, 2017
    Applicant: Magic Leap, Inc.
    Inventors: Xiaolin Wei, Yifu Zhang
  • Patent number: 9674506
    Abstract: A system that incorporates teachings of the exemplary embodiments may include, for example, means for generating a disparity map based on a depth map, means for determining accuracy of pixels in the depth map where the determining means identifies the pixels as either accurate or inaccurate based on a confidence map and the disparity map, and means for providing an adjusted depth map where the providing means adjusts inaccurate pixels of the depth map using a cost function associated with the inaccurate pixels. Other embodiments are disclosed.
    Type: Grant
    Filed: September 3, 2015
    Date of Patent: June 6, 2017
    Assignee: AT&T Intellectual Property I, L.P.
    Inventors: Chao Tian, Vinay Anant Vaishampayan, Yifu Zhang