Patents by Inventor Shuochen Su
Shuochen Su has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20230280468Abstract: A projector for illuminating a target area is presented. The projector includes an array of emitters having a plurality of subarrays and an optical assembly. Each subarray includes one or more independently addressable channels emitting light in accordance with emission instructions. At least two of the subarrays are adjacent to each other and do not overlap. The optical assembly is configured to tile portions of the emitted light to form a light pattern for projection to a target area. The light pattern has a first plurality of sections and a second plurality of sections. Each section of the first plurality represents a first respective portion of the light pattern emitted from a corresponding subarray. Each section of the second plurality represents a second respective portion of the light pattern formed by tiling light emitted from two or more of the subarrays.Type: ApplicationFiled: May 19, 2022Publication date: September 7, 2023Inventors: Zhaoming Zhu, Michael Hall, Augusto Ronchini Ximenes, Shuochen Su, Charles Patrick Saunders, Gregory Cohoon
-
Patent number: 11740075Abstract: A depth camera assembly (DCA) determines depth information. The DCA projects a dynamic structured light pattern into a local area and captures images including a portion of the dynamic structured light pattern. The DCA determines regions of interest in which it may be beneficial to increase or decrease an amount of texture added to the region of interest using the dynamic structured light pattern. For example, the DCA may identify the regions of interest based on contrast values calculated using a contrast algorithm, or based on the parameters received from a mapping server including a virtual model of the local area. The DCA may selectively increase or decrease an amount of texture added by the dynamic structured light pattern in portions of the local area. By selectively controlling portions of the dynamic structured light pattern, the DCA may decrease power consumption and/or increase the accuracy of depth sensing measurements.Type: GrantFiled: October 18, 2021Date of Patent: August 29, 2023Assignee: Meta Platforms Technologies, LLCInventors: Michael Hall, Xinqiao Liu, Zhaoming Zhu, Rajesh Lachhmandas Chhabria, Huixuan Tang, Shuochen Su, Zihe Gao
-
Publication number: 20230260222Abstract: A binary depth mask model is trained for use in occlusion within a mixed reality application. The training leverages information about virtual object positions and depths that is inherently available to the system as part of rendering of virtual objects. Training is performed on a set of stereo images, a set of binary depth masks corresponding to the stereo images, and a depth value against which object depth is evaluated. Given this input, the training outputs the binary depth mask model, which when given a stereo image as input outputs a depth binary depth mask indicating which pixels of the stereo image are nearer, or father away, than the depth value. The depth mask model can be applied in real time to handle occlusion operations when compositing a given real-world stereo image with virtual objects.Type: ApplicationFiled: December 22, 2022Publication date: August 17, 2023Inventors: Brevin Tilmon, Shuochen Su, Michael Hall
-
Patent number: 11703323Abstract: A depth estimation system is described capable of determining depth information using two images from two cameras. A first camera captures a first image and a second camera captures a second image, both images including a plurality of light channels. A scan direction is selected from a plurality of scan directions. For the selected scan direction, along each of a plurality of scanlines, the system compares pixels from the first image to pixels from the second image. The comparison is based on calculating a census transform for each pixel in the first image and a census transform for each pixel in the second image. This comparison is used to determine a stereo correspondence between the pixels in the first image and the pixels in the second image. The system generates a depth map based on the stereo correspondence.Type: GrantFiled: April 14, 2021Date of Patent: July 18, 2023Assignee: Meta Platforms Technologies, LLCInventors: Michael Hall, Xinqiao Liu, Zhaoming Zhu, Rajesh Lachhmandas Chhabria, Huixuan Tang, Shuochen Su
-
Patent number: 11601607Abstract: An electronic device includes one or more processors and memory storing instructions for execution by the one or more processors. The stored instructions include instructions for: receiving infrared image information for a three-dimensional area; receiving non-infrared image information for the same three-dimensional area; performing nonlinear intensity adjustment for the received infrared image information; performing nonlinear intensity adjustment for the received non-infrared image information; blending the intensity-adjusted infrared image information and the intensity-adjusted non-infrared image information to obtain a merged image information; and providing the merged image information for determining a depth map. Also disclosed are a corresponding method performed by the electronic device and a computer readable storage medium storing instructions for execution by one or more processors of an electronic device.Type: GrantFiled: October 13, 2020Date of Patent: March 7, 2023Assignee: META PLATFORMS TECHNOLOGIES, LLCInventors: Rajesh Lachhmandas Chhabria, Shuochen Su, Monica Stewart, David Vakrat, Shengqiong Xie, Michael Hall
-
Patent number: 11480684Abstract: A direct time of flight depth system includes an addressable illumination source, an active depth sensor, and a controller. The addressable illumination source includes an array of emitters and an optical assembly that is used to generate an array of dots emitted into a local area. Each emitter is independently addressable, allowing selective illumination of different portions of the local area. The addressable illumination source is aligned with the active depth sensor so each dot maps to a corresponding macropixel (e.g., 4×4 array of SPADs) on the active depth sensor. Data from the active depth sensor is readout and used to determine depth information.Type: GrantFiled: December 30, 2020Date of Patent: October 25, 2022Assignee: Meta Platforms Technologies, LLCInventors: Augusto Ronchini Ximenes, Michael Hall, Zhaoming Zhu, Shuochen Su
-
Publication number: 20220036571Abstract: A depth camera assembly (DCA) determines depth information. The DCA projects a dynamic structured light pattern into a local area and captures images including a portion of the dynamic structured light pattern. The DCA determines regions of interest in which it may be beneficial to increase or decrease an amount of texture added to the region of interest using the dynamic structured light pattern. For example, the DCA may identify the regions of interest based on contrast values calculated using a contrast algorithm, or based on the parameters received from a mapping server including a virtual model of the local area. The DCA may selectively increase or decrease an amount of texture added by the dynamic structured light pattern in portions of the local area. By selectively controlling portions of the dynamic structured light pattern, the DCA may decrease power consumption and/or increase the accuracy of depth sensing measurements.Type: ApplicationFiled: October 18, 2021Publication date: February 3, 2022Inventors: Michael Hall, Xinqiao Liu, Zhaoming Zhu, Rajesh Lachhmandas Chhabria, Huixuan Tang, Shuochen Su, Zihe Gao
-
Publication number: 20220030183Abstract: An electronic device includes one or more processors and memory storing instructions for execution by the one or more processors. The stored instructions include instructions for: receiving infrared image information for a three-dimensional area; receiving non-infrared image information for the same three-dimensional area; performing nonlinear intensity adjustment for the received infrared image information; performing nonlinear intensity adjustment for the received non-infrared image information; blending the intensity-adjusted infrared image information and the intensity-adjusted non-infrared image information to obtain a merged image information; and providing the merged image information for determining a depth map. Also disclosed are a corresponding method performed by the electronic device and a computer readable storage medium storing instructions for execution by one or more processors of an electronic device.Type: ApplicationFiled: October 13, 2020Publication date: January 27, 2022Inventors: Rajesh Lachhmandas CHHABRIA, Shuochen SU, Monica STEWART, David VAKRAT, Shengqiong XIE, Michael HALL
-
Publication number: 20220028102Abstract: A method of determining confidence in stereo matching includes receiving left image information; receiving right image information; determining a stereo matching difference profile between the left image information and the right image information; and determining one or more confidence values of stereo matching consistency by applying a predefined classifier to the stereo matching difference profile. Associated electronic devices and computer readable storage media are also disclosed.Type: ApplicationFiled: October 19, 2020Publication date: January 27, 2022Inventors: Rajesh Lachhmandas CHHABRIA, Shuochen SU
-
Publication number: 20220028103Abstract: A depth estimation system is described capable of determining depth information using two images from two cameras. A first camera captures a first image and a second camera captures a second image, both images including a plurality of light channels. A scan direction is selected from a plurality of scan directions. For the selected scan direction, along each of a plurality of scanlines, the system compares pixels from the first image to pixels from the second image. The comparison is based on calculating a census transform for each pixel in the first image and a census transform for each pixel in the second image. This comparison is used to determine a stereo correspondence between the pixels in the first image and the pixels in the second image. The system generates a depth map based on the stereo correspondence.Type: ApplicationFiled: April 14, 2021Publication date: January 27, 2022Inventors: Michael Hall, Xinqiao Liu, Zhaoming Zhu, Rajesh Lachhmandas Chhabria, Huixuan Tang, Shuochen Su
-
Publication number: 20210396886Abstract: A direct time of flight depth system includes an addressable illumination source, an active depth sensor, and a controller. The addressable illumination source includes an array of emitters and an optical assembly that is used to generate an array of dots emitted into a local area. Each emitter is independently addressable, allowing selective illumination of different portions of the local area. The addressable illumination source is aligned with the active depth sensor so each dot maps to a corresponding macropixel (e.g., 4×4 array of SPADs) on the active depth sensor. Data from the active depth sensor is readout and used to determine depth information.Type: ApplicationFiled: December 30, 2020Publication date: December 23, 2021Inventors: Augusto Ronchini Ximenes, Michael Hall, Zhaoming Zhu, Shuochen Su
-
Patent number: 11195291Abstract: A depth camera assembly (DCA) optimizes illumination and image capture of a local area to generate depth information of the local area. The DCA determines depth information for a first portion of the local area viewable at a first pose. The DCA is moved from the first pose to a second pose, where a second portion of the local area is viewable and overlaps with the first portion. The overlapping region is not illuminated by the DCA. A non-overlapping portion of the second portion is illuminated, captured, and depth information determined.Type: GrantFiled: February 19, 2020Date of Patent: December 7, 2021Assignee: Facebook Technologies, LLCInventors: Fengqiang Li, Zihe Gao, Michael Hall, Zhaoming Zhu, Shuochen Su, Huixuan Tang, Xinqiao Liu, Nicholas Daniel Trail
-
Publication number: 20210366142Abstract: A depth camera assembly (DCA) determines depth information for a local area. The DCA includes a plurality of cameras and at least one illuminator. The DCA dynamically determines depth sensing modes (e.g., passive stereo, active stereo, structured stereo) based in part on the surrounding environment and/or user activity. The DCA uses the depth information to update a depth model describing the local area. The DCA may determine that a portion of the depth information associated with some of portion of the local area is not accurate. The DCA may then select a different depth sensing mode for the portion of the local area and update the depth model with the additional depth information. In some embodiments, the DCA may update the depth model by utilizing a machine learning model to generate a refined depth model.Type: ApplicationFiled: May 25, 2021Publication date: November 25, 2021Inventors: Nadav Grossinger, Michael Hall, Robert Hasbun, Niv Kantor, Arthur Benjamin Nieuwoudt, Shuochen Su, Andrey Tovchigrechko, Yue Zhao
-
Patent number: 11182914Abstract: A depth camera assembly (DCA) determines depth information. The DCA projects a dynamic structured light pattern into a local area and captures images including a portion of the dynamic structured light pattern. The DCA determines regions of interest in which it may be beneficial to increase or decrease an amount of texture added to the region of interest using the dynamic structured light pattern. For example, the DCA may identify the regions of interest based on contrast values calculated using a contrast algorithm, or based on the parameters received from a mapping server including a virtual model of the local area. The DCA may selectively increase or decrease an amount of texture added by the dynamic structured light pattern in portions of the local area. By selectively controlling portions of the dynamic structured light pattern, the DCA may decrease power consumption and/or increase the accuracy of depth sensing measurements.Type: GrantFiled: May 17, 2019Date of Patent: November 23, 2021Assignee: Facebook Technologies, LLCInventors: Michael Hall, Xinqiao Liu, Zhaoming Zhu, Rajesh Lachhmandas Chhabria, Huixuan Tang, Shuochen Su, Zihe Gao
-
Patent number: 11010911Abstract: A depth estimation system is described capable of determining depth information using two images from two cameras. A first camera captures a first image and a second camera captures a second image, both images including a plurality of light channels. In a first light channel of the plurality of light channels, the system calculates a census transform for each pixel of the first image and a census transform for each pixel of the second image. In a second light channel of the plurality of light channels, the system calculates a census transform for each pixel of the first image and a census transform for each pixel of the second image. The system generates a depth map based in part on the census transforms for each pixel of the first image and the second image in the first light channel and in the second light channel.Type: GrantFiled: May 21, 2019Date of Patent: May 18, 2021Assignee: Facebook Technologies, LLCInventors: Michael Hall, Xinqiao Liu, Zhaoming Zhu, Rajesh Lachhmandas Chhabria, Huixuan Tang, Shuochen Su
-
Patent number: 10972715Abstract: A depth camera assembly (DCA) determines depth information within a local area. The DCA may selectively process a subset of data captured by an imaging sensor and obtained from the imaging sensor, such as pixels corresponding to a region of interest, for depth information. Alternatively, the DCA may limit retrieval of data from the imaging sensor to pixels corresponding to the region of interest from the imaging sensor for processing for depth information. The depth processing may include a semi-global match (SGM) algorithm, and the DCA adjusts a number of neighboring pixels used for determining depth information for a specific pixel based on one or more criteria. In some embodiments, the DCA performs the depth processing by analyzing images from different image sensors using left to right and right to left correspondence checks that are performed in parallel.Type: GrantFiled: May 21, 2019Date of Patent: April 6, 2021Assignee: Facebook Technologies, LLCInventors: Michael Hall, Xinqiao Liu, Zhaoming Zhu, Rajesh Lachhmandas Chhabria, Huixuan Tang, Shuochen Su
-
Patent number: 10929997Abstract: A depth camera assembly (DCA) determines depth information within a local area by capturing images of the local area including a local region using a plurality of imaging sensors. The local region is represented by a first set of pixels in each captured image. For each image, the DCA identifies the first set of pixels corresponding to the surface in the local region and determines a depth measurement from the DCA to the local region by comparing the first set of pixels from images captured by different imaging sensors. To determine depth measurements for second sets of pixels neighboring the first set of pixels, the DCA selectively propagates depth information from the first set of pixels to second sets of pixels satisfying one or more criteria (e.g., satisfying a threshold saturation measurement or a threshold contrast measurement).Type: GrantFiled: May 21, 2019Date of Patent: February 23, 2021Assignee: Facebook Technologies, LLCInventors: Michael Hall, Xinqiao Liu, Zhaoming Zhu, Rajesh Lachhmandas Chhabria, Huixuan Tang, Shuochen Su, Zihe Gao
-
Patent number: 10755173Abstract: Methods and systems are provided for deblurring images. A neural network is trained where the training includes selecting a central training image from a sequence of blurred images. An earlier training image and a later training image are selected based on the earlier training image preceding the central training image in the sequence and the later training image following the central training image in the sequence and based on proximity of the images to the central training image in the sequence. A training output image is generated by the neural network from the central training image, the earlier training image, and the later training image. Similarity is evaluated between the training output image and a reference image. The neural network is modified based on the evaluated similarity. The trained neural network is used to generate a deblurred output image from a blurry input image.Type: GrantFiled: December 4, 2019Date of Patent: August 25, 2020Assignee: ADOBE INC.Inventors: Oliver Wang, Jue Wang, Shuochen Su
-
Publication number: 20200111004Abstract: Methods and systems are provided for deblurring images. A neural network is trained where the training includes selecting a central training image from a sequence of blurred images. An earlier training image and a later training image are selected based on the earlier training image preceding the central training image in the sequence and the later training image following the central training image in the sequence and based on proximity of the images to the central training image in the sequence. A training output image is generated by the neural network from the central training image, the earlier training image, and the later training image. Similarity is evaluated between the training output image and a reference image. The neural network is modified based on the evaluated similarity. The trained neural network is used to generate a deblurred output image from a blurry input image.Type: ApplicationFiled: December 4, 2019Publication date: April 9, 2020Inventors: OLIVER WANG, JUE WANG, SHUOCHEN SU
-
Patent number: 10534998Abstract: Methods and systems are provided for deblurring images. A neural network is trained where the training includes selecting a central training image from a sequence of blurred images. An earlier training image and a later training image are selected based on the earlier training image preceding the central training image in the sequence and the later training image following the central training image in the sequence and based on proximity of the images to the central training image in the sequence. A training output image is generated by the neural network from the central training image, the earlier training image, and the later training image. Similarity is evaluated between the training output image and a reference image. The neural network is modified based on the evaluated similarity. The trained neural network is used to generate a deblurred output image from a blurry input image.Type: GrantFiled: April 10, 2019Date of Patent: January 14, 2020Assignee: Adobe Inc.Inventors: Oliver Wang, Jue Wang, Shuochen Su