Patents by Inventor Muge Wang

Muge Wang has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Patent number: 11968471
    Abstract: Embodiments relate to extracting features from images, such as by identifying keypoints and generating keypoint descriptors of the keypoints. An apparatus includes a pyramid image generator circuit, a keypoint descriptor generator circuit, and a pyramid image buffer. The pyramid image generator circuit generates an image pyramid from an input image. The keypoint descriptor generator circuit processes the pyramid images for keypoint descriptor generation. The pyramid image buffer stores different portions of the pyramid images generated by the pyramid image generator circuit at different times and provides the stored portions of the pyramid images to the keypoint descriptor generator circuit for keypoint descriptor generation. When first portions of the pyramid images are no longer needed for the keypoint descriptor generation, the first portions are removed from the pyramid image buffer to provide space for second portions of the pyramid images that are needed for the keypoint descriptor generation.
    Type: Grant
    Filed: March 8, 2021
    Date of Patent: April 23, 2024
    Assignee: APPLE INC.
    Inventors: David R. Pope, Liran Fishel, Assaf Metuki, Muge Wang
  • Patent number: 11810266
    Abstract: Embodiments relate to generating keypoint descriptors of the keypoints using a sub-scale refinement and a sample pattern radius adjustment. An apparatus includes a sub-pixel refiner circuit and a keypoint descriptor generator circuit. The sub-pixel refiner circuit determines a keypoint scale value for a scale dimension of a keypoint in an image pyramid by performing an interpolation of response map (RM) pixel values of a pixel block of RM images defined around the keypoint. The keypoint descriptor generator circuit determines sample scales of the image pyramid based on the keypoint scale value and determines a radius value for each sample scale based on the keypoint scale value. The keypoint descriptor generator circuit samples patches of pixel values at the sample scales using the radius value for each sample scale to generate a keypoint descriptor of the keypoint.
    Type: Grant
    Filed: March 19, 2021
    Date of Patent: November 7, 2023
    Assignee: APPLE INC.
    Inventors: Muge Wang, Assaf Metuki
  • Patent number: 11494880
    Abstract: Embodiments relate to generating an image pyramid for feature extraction. A pyramid image generator circuit includes a first image buffer that stores image data at a first octave, a first blur filter circuit, a first spatial filter circuit, and a first decimator circuit. The first blur filter circuit generates a first pyramid image for a first scale of the first octave by applying a first amount of smoothing to the first image data stored in the first image buffer. The first spatial filter circuit and the first decimator generate second image data of a second octave that is higher than the first octave by applying a smoothing and a decimation to the first image data stored in the first image buffer. The first spatial filter circuit begins processing the first image data before the first blur filter circuit begins to process the first image data.
    Type: Grant
    Filed: March 19, 2021
    Date of Patent: November 8, 2022
    Assignee: Apple Inc.
    Inventors: David R. Pope, Assaf Metuki, Muge Wang
  • Publication number: 20220301110
    Abstract: Embodiments relate to generating an image pyramid for feature extraction. A pyramid image generator circuit includes a first image buffer that stores image data at a first octave, a first blur filter circuit, a first spatial filter circuit, and a first decimator circuit. The first blur filter circuit generates a first pyramid image for a first scale of the first octave by applying a first amount of smoothing to the first image data stored in the first image buffer. The first spatial filter circuit and the first decimator generate second image data of a second octave that is higher than the first octave by applying a smoothing and a decimation to the first image data stored in the first image buffer. The first spatial filter circuit begins processing the first image data before the first blur filter circuit begins to process the first image data.
    Type: Application
    Filed: March 19, 2021
    Publication date: September 22, 2022
    Inventors: David R. Pope, Assaf Metuki, Muge Wang
  • Publication number: 20220301104
    Abstract: Embodiments relate to generating keypoint descriptors of the keypoints using a sub-scale refinement and a sample pattern radius adjustment. An apparatus includes a sub-pixel refiner circuit and a keypoint descriptor generator circuit. The sub-pixel refiner circuit determines a keypoint scale value for a scale dimension of a keypoint in an image pyramid by performing an interpolation of response map (RM) pixel values of a pixel block of RM images defined around the keypoint. The keypoint descriptor generator circuit determines sample scales of the image pyramid based on the keypoint scale value and determines a radius value for each sample scale based on the keypoint scale value. The keypoint descriptor generator circuit samples patches of pixel values at the sample scales using the radius value for each sample scale to generate a keypoint descriptor of the keypoint.
    Type: Application
    Filed: March 19, 2021
    Publication date: September 22, 2022
    Inventors: Muge Wang, Assaf Metuki
  • Publication number: 20220286604
    Abstract: Embodiments relate to extracting features from images, such as by identifying keypoints and generating keypoint descriptors of the keypoints. An apparatus includes a pyramid image generator circuit, a keypoint descriptor generator circuit, and a pyramid image buffer. The pyramid image generator circuit generates an image pyramid from an input image. The keypoint descriptor generator circuit processes the pyramid images for keypoint descriptor generation. The pyramid image buffer stores different portions of the pyramid images generated by the pyramid image generator circuit at different times and provides the stored portions of the pyramid images to the keypoint descriptor generator circuit for keypoint descriptor generation. When first portions of the pyramid images are no longer needed for the keypoint descriptor generation, the first portions are removed from the pyramid image buffer to provide space for second portions of the pyramid images that are needed for the keypoint descriptor generation.
    Type: Application
    Filed: March 8, 2021
    Publication date: September 8, 2022
    Inventors: David R. Pope, Liran Fishel, Assaf Metuki, Muge Wang
  • Patent number: 11330296
    Abstract: Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, a video encoding system includes image processing circuitry configured to receive source image data and derive full-resolution image data and low-resolution image data from the source image data. The video encoding system also includes a low resolution pipeline configured to receive the low-resolution image data and determine one or more low resolution inter prediction modes based on the low-resolution image data. Furthermore, the video encoding system includes a main pipeline configured to encode the full-resolution image data based on the one or more low resolution inter prediction modes.
    Type: Grant
    Filed: September 14, 2020
    Date of Patent: May 10, 2022
    Assignee: Apple Inc.
    Inventors: Guy Côté, Athanasios Leontaris, Muge Wang
  • Publication number: 20220086485
    Abstract: Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, a video encoding system includes image processing circuitry configured to receive source image data and derive full-resolution image data and low-resolution image data from the source image data. The video encoding system also includes a low resolution pipeline configured to receive the low-resolution image data and determine one or more low resolution inter prediction modes based on the low-resolution image data. Furthermore, the video encoding system includes a main pipeline configured to encode the full-resolution image data based on the one or more low resolution inter prediction modes.
    Type: Application
    Filed: September 14, 2020
    Publication date: March 17, 2022
    Inventors: Guy Côté, Athanasios Leontaris, Muge Wang
  • Patent number: 11082606
    Abstract: Determining a focus setting includes determining a plurality of regions of interest in a view of a scene, and, for each of the plurality of regions of interest, obtaining a set of image data for each of multiple focal positions, and then applying focus filters to the set of image data for each of the plurality of focal positions for each of the regions of interest to obtain a set of focus scores, i.e., a focus score for each focus filter applied to the set of image data for each of the focal positions. Further, determining a confidence value associated with each of the sets of focus scores, selecting a subset of the sets of focus scores based on the confidence values associated with each of the sets of focus scores, and determining a focus setting for the scene based on the selected subset of the focus scores.
    Type: Grant
    Filed: September 11, 2018
    Date of Patent: August 3, 2021
    Assignee: Apple Inc.
    Inventors: Mark N. Gamadia, Yingjun Bai, Muge Wang
  • Patent number: 10997736
    Abstract: Embodiments relate to a normalized cross correlation (NCC) circuit that can perform a normalized cross correlation between input patch data and kernel data. An interface circuit of an image signal processor receives input patch data from a source. Input patch data is data that represents a portion of a frame of image data from the source. The NCC circuit includes a filtering circuit and a normalization circuit. The filtering circuit receives the input patch data from the interface circuit and performs a convolution on the received input patch data or processed patch data derived from the input patch data with kernel data to produce convolution output data. The normalization circuit computes a normalized score output based on the convolution output data and the kernel data. The normalized score output includes normalization scores for each location of the convolution output data.
    Type: Grant
    Filed: August 10, 2018
    Date of Patent: May 4, 2021
    Assignee: Apple Inc.
    Inventors: Muge Wang, Junji Sugisawa
  • Patent number: 10855964
    Abstract: Embodiments relate to generation of hue maps for highlight recovery of an input image. An image having a plurality of color channels is obtained at a first resolution lower than a resolution of the input image. A hue for each color channel for each pixel is determined, using a pixel value for that color channel and pixel values for the plurality of color channels in the first image. Weights are determined for each pixel for each color channel, based on hues for the pixel and pixel values for the pixel in the first image. A plurality of candidate hue maps are generated, based on the weights and pixel values in the first image in a patch surrounding the pixel for the plurality of color channels.
    Type: Grant
    Filed: August 29, 2018
    Date of Patent: December 1, 2020
    Assignee: Apple Inc.
    Inventors: Frederic Cao, Touraj Tajbakhsh, Muge Wang
  • Patent number: 10747843
    Abstract: Embodiments of the present disclosure relate to a configurable convolution engine that receives configuration information to perform convolution or its variant operations on streaming input data of various formats. To process streaming input data, input data of multiple channels are received and stored in an input buffer circuit in an interleaved manner. Data values of the interleaved input data are retrieved and forwarded to multiplier circuits where multiplication with a corresponding filter element of a kernel is performed. Varying number of kernels with different sizes and sparsity can also be used for the convolution operations.
    Type: Grant
    Filed: February 14, 2020
    Date of Patent: August 18, 2020
    Assignee: Apple Inc.
    Inventors: Suk Hwan Lim, Junji Sugisawa, Muge Wang
  • Patent number: 10692177
    Abstract: Embodiments relate to a first demosaicing circuit and a second demosaicing circuit that can perform demosaicing of image data. The first demosaicing circuit processes received image data to generate a first demosaiced image for obtaining statistic information on the received image data. The second demosaicing circuit performs demosaicing of the received image data to generate a second demosaiced image. A processing circuit pipeline performs at least one of resampling, noise processing, color processing and output rescaling performed on the second demosaiced image based on the statistics information obtained from the first demosaiced image.
    Type: Grant
    Filed: August 10, 2018
    Date of Patent: June 23, 2020
    Assignee: Apple Inc.
    Inventors: Muge Wang, David R. Pope
  • Patent number: 10685421
    Abstract: Embodiments relate to a configurable convolution engine that receives configuration information to perform convolution and other deep machine learning operations on streaming input data of various formats. The convolution engine may include two convolution circuits that each generate a stream of values by applying convolution kernels to input data. The stream of values may each define one or more channels of image data. A channel merge circuit combines the streams of values from each convolution circuit in accordance with a selected mode of operation. In one mode, the first and second streams from the convolution circuits are merged into an output stream having the combined channels of the first and second streams in an interleaved manner. In another mode, the first stream from the first convolution circuit is fed into the input of the second convolution circuit.
    Type: Grant
    Filed: October 25, 2019
    Date of Patent: June 16, 2020
    Assignee: Apple Inc.
    Inventors: Sung Hee Park, Muge Wang, Junji Sugisawa
  • Publication number: 20200184000
    Abstract: Embodiments of the present disclosure relate to a configurable convolution engine that receives configuration information to perform convolution or its variant operations on streaming input data of various formats. To process streaming input data, input data of multiple channels are received and stored in an input buffer circuit in an interleaved manner. Data values of the interleaved input data are retrieved and forwarded to multiplier circuits where multiplication with a corresponding filter element of a kernel is performed. Varying number of kernels with different sizes and sparsity can also be used for the convolution operations.
    Type: Application
    Filed: February 14, 2020
    Publication date: June 11, 2020
    Inventors: Suk Hwan Lim, Junji Sugisawa, Muge Wang
  • Publication number: 20200167889
    Abstract: Embodiments relate to a configurable convolution engine that receives configuration information to perform convolution and other deep machine learning operations on streaming input data of various formats. The convolution engine may include two convolution circuits that each generate a stream of values by applying convolution kernels to input data. The stream of values may each define one or more channels of image data. A channel merge circuit combines the streams of values from each convolution circuit in accordance with a selected mode of operation. In one mode, the first and second streams from the convolution circuits are merged into an output stream having the combined channels of the first and second streams in an interleaved manner. In another mode, the first stream from the first convolution circuit is fed into the input of the second convolution circuit.
    Type: Application
    Filed: October 25, 2019
    Publication date: May 28, 2020
    Inventors: Sung Hee Park, Muge Wang, Junji Sugisawa
  • Patent number: 10606918
    Abstract: Embodiments of the present disclosure relate to a configurable convolution engine that receives configuration information to perform convolution or its variant operations on streaming input data of various formats. To process streaming input data, input data of multiple channels are received and stored in an input buffer circuit in an interleaved manner. Data values of the interleaved input data are retrieved and forwarded to multiplier circuits where multiplication with a corresponding filter element of a kernel is performed. Varying number of kernels with different sizes and sparsity can also be used for the convolution operations.
    Type: Grant
    Filed: October 25, 2019
    Date of Patent: March 31, 2020
    Assignee: Apple Inc.
    Inventors: Suk Hwan Lim, Junji Sugisawa, Muge Wang
  • Publication number: 20200077066
    Abstract: Embodiments relate to generation of hue maps for highlight recovery of an input image. An image having a plurality of color channels is obtained at a first resolution lower than a resolution of the input image. A hue for each color channel for each pixel is determined, using a pixel value for that color channel and pixel values for the plurality of color channels in the first image. Weights are determined for each pixel for each color channel, based on hues for the pixel and pixel values for the pixel in the first image. A plurality of candidate hue maps are generated, based on the weights and pixel values in the first image in a patch surrounding the pixel for the plurality of color channels.
    Type: Application
    Filed: August 29, 2018
    Publication date: March 5, 2020
    Inventors: Frederic Cao, Touraj Tajbakhsh, Muge Wang
  • Publication number: 20200057789
    Abstract: Embodiments of the present disclosure relate to a configurable convolution engine that receives configuration information to perform convolution or its variant operations on streaming input data of various formats. To process streaming input data, input data of multiple channels are received and stored in an input buffer circuit in an interleaved manner. Data values of the interleaved input data are retrieved and forwarded to multiplier circuits where multiplication with a corresponding filter element of a kernel is performed. Varying number of kernels with different sizes and sparsity can also be used for the convolution operations.
    Type: Application
    Filed: October 25, 2019
    Publication date: February 20, 2020
    Inventors: Suk Hwan Lim, Junji Sugisawa, Muge Wang
  • Patent number: 10567787
    Abstract: Embodiments of the present disclosure relate to autofocusing of images using motion vectors generated by an image signal processor of a device. An image being processed may include one or more motion detection windows associated with a motion vector as well as one or more autofocus windows. An autofocus window that follows a motion detection window by at least a threshold vertical distance may be selected, for example, to account for a period of time or latency for determining a motion vector of the motion detection window. The device may perform autofocusing by shifting location of the selected autofocus window.
    Type: Grant
    Filed: August 10, 2018
    Date of Patent: February 18, 2020
    Assignee: Apple Inc.
    Inventors: Muge Wang, D. Amnon Silverstein