Patents by Inventor Sudeng Hu

Sudeng Hu has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Patent number: 11677934
    Abstract: In an example method, a system receives a plurality of frames of a video, and generates a data structure representing the video and representing a plurality of temporal layers. Generating the data structure includes: (i) determining a plurality of quality levels for presenting the video, where each of the quality levels corresponds to a different respective sampling period for sampling the frames of the video, (ii) assigning, based on the sampling periods, each of the frames to a respective one of the temporal layers of the data structure, and (iii) indicating, in the data structure, one or more relationships between (a) at least one the frames assigned to at least one of the temporal layers of the data structure, and (b) at least another one of the frames assigned to at least another one of the temporal layers of the data structure. Further, the system outputs the data structure.
    Type: Grant
    Filed: September 24, 2021
    Date of Patent: June 13, 2023
    Assignee: Apple Inc.
    Inventors: Sudeng Hu, David L. Biderman, Christopher M. Garrido, Hsi-Jung Wu, Xiaosong Zhou, Dazhong Zhang, Jinbo Qiu, Karthick Santhanam, Hang Yuan, Joshua L. Hare, Luciano M. Verger, Kevin Arthur Robertson, Sasanka Vemuri
  • Publication number: 20230098082
    Abstract: In an example method, a system receives a plurality of frames of a video, and generates a data structure representing the video and representing a plurality of temporal layers. Generating the data structure includes: (i) determining a plurality of quality levels for presenting the video, where each of the quality levels corresponds to a different respective sampling period for sampling the frames of the video, (ii) assigning, based on the sampling periods, each of the frames to a respective one of the temporal layers of the data structure, and (iii) indicating, in the data structure, one or more relationships between (a) at least one the frames assigned to at least one of the temporal layers of the data structure, and (b) at least another one of the frames assigned to at least another one of the temporal layers of the data structure. Further, the system outputs the data structure.
    Type: Application
    Filed: September 24, 2021
    Publication date: March 30, 2023
    Inventors: Sudeng Hu, David L. Biderman, Christopher M. Garrido, Hsi-Jung Wu, Xiaosong Zhou, Dazhong Zhang, Jinbo Qiu, Karthick Santhanam, Hang Yuan, Joshua L. Hare, Luciano M. Verger, Kevin Arthur Robertson, Sasanka Vemuri
  • Patent number: 11553200
    Abstract: Techniques are disclosed for coding and decoding video data using object recognition and object modeling as a basis of coding and error recovery. A video decoder may decode coded video data received from a channel. The video decoder may perform object recognition on decoded video data obtained therefrom, and, when an object is recognized in the decoded video data, the video decoder may generate a model representing the recognized object. It may store data representing the model locally. The video decoder may communicate the model data to an encoder, which may form a basis of error mitigation and recovery. The video decoder also may monitor deviation patterns in the object model and associated patterns in audio content; if/when video decoding is suspended due to operational errors, the video decoder may generate simulated video data by analyzing audio data received during the suspension period and developing video data from the data model and deviation(s) associated with patterns detected from the audio data.
    Type: Grant
    Filed: May 11, 2020
    Date of Patent: January 10, 2023
    Assignee: APPLE INC.
    Inventors: Xing Wen, Dazhong Zhang, Peikang Song, Xiaosong Zhou, Sudeng Hu, Hsi-Jung Wu, Jae Hoon Kim
  • Patent number: 11184415
    Abstract: Techniques presented herein provide an improved relay user experience and improved management of scarce computing and network resources as the number of relay endpoints increases. A sourcing endpoint device may generate a media feed, such as video and/or audio feed, representing contribution from a conference participant. The sourcing endpoint device may generate a priority value for the media feed, and the priority value may be transmitted to other members of the relay along with the input feed. Priority values of the different relay participants may be used by other devices, for example, intermediate servers or receiving endpoint devices, to manage aspects of the relay. For example, a relay server may prune streams from select endpoint devices based on relative priority values received from those devices. Alternatively, receiving endpoint devices may alter presentation of received feeds based on their associated priority values.
    Type: Grant
    Filed: May 7, 2019
    Date of Patent: November 23, 2021
    Assignee: Apple Inc.
    Inventors: Christopher M. Garrido, Dazhong Zhang, Karthick Santhanam, Patrick Miauton, Xiaoxiao Zheng, Bess Chan, Peter Shiang, Sudeng Hu, Peikang Song, Xiaosong Zhou
  • Patent number: 11109042
    Abstract: Systems and methods for coding a video to be overlaid by annotations are devised. A motion compensated predictive coding is employed, wherein coding parameters of video pixel blocks are determined based on the pixel blocks' relation to the annotations. A decoder decodes the video and annotates it based on metadata, obtained from the coder or other sources, describing the annotations' appearance and rendering mode.
    Type: Grant
    Filed: May 23, 2019
    Date of Patent: August 31, 2021
    Assignee: Apple Inc.
    Inventors: Sudeng Hu, Xing Wen, Jae Hoon Kim, Peikang Song, Hang Yuan, Dazhong Zhang, Xiaosong Zhou, Hsi-Jung Wu, Christopher Garrido, Ming Jin, Patrick Miauton, Karthick Santhanam
  • Patent number: 11025933
    Abstract: Techniques are disclosed for managing memory allocations when coding video data according to multiple codec configurations. According to these techniques, devices may negotiate parameters of a coding session that include parameters of a plurality of different codec configurations that may be used during the coding session. A device may estimate sizes of decoded picture buffers for each of the negotiated codec configurations and allocate in its memory a portion of memory sized according to a largest size of the estimated decoded picture buffers. Thereafter, the devices may exchange coded video data. The exchange may involve decoding coded data of reference pictures and storing the decoded reference pictures in the allocated memory. During the coding session, the devices may toggle among the different negotiated codec configurations. As they do, reallocations of memory may be avoided.
    Type: Grant
    Filed: May 3, 2017
    Date of Patent: June 1, 2021
    Assignee: Apple Inc.
    Inventors: Dazhong Zhang, Hang Yuan, Peikang Song, Jae Hoon Kim, Xing Wen, Sudeng Hu, Xiaosong Zhou, Chris Y. Chung, Hsi-Jung Wu
  • Publication number: 20200389662
    Abstract: Techniques are disclosed for coding and decoding video data using object recognition and object modeling as a basis of coding and error recovery. A video decoder may decode coded video data received from a channel. The video decoder may perform object recognition on decoded video data obtained therefrom, and, when an object is recognized in the decoded video data, the video decoder may generate a model representing the recognized object. It may store data representing the model locally. The video decoder may communicate the model data to an encoder, which may form a basis of error mitigation and recovery. The video decoder also may monitor deviation patterns in the object model and associated patterns in audio content; if/when video decoding is suspended due to operational errors, the video decoder may generate simulated video data by analyzing audio data received during the suspension period and developing video data from the data model and deviation(s) associated with patterns detected from the audio data.
    Type: Application
    Filed: May 11, 2020
    Publication date: December 10, 2020
    Inventors: Xing Wen, Dazhong Zhang, Peikang Song, Xiaosong Zhou, Sudeng Hu, Hsi-Jung Wu, Jae Hoon Kim
  • Patent number: 10652567
    Abstract: Techniques are disclosed for coding and decoding video data using object recognition and object modeling as a basis of coding and error recovery. A video decoder may decode coded video data received from a channel. The video decoder may perform object recognition on decoded video data obtained therefrom, and, when an object is recognized in the decoded video data, the video decoder may generate a model representing the recognized object. It may store data representing the model locally. The video decoder may communicate the model data to an encoder, which may form a basis of error mitigation and recovery. The video decoder also may monitor deviation patterns in the object model and associated patterns in audio content; if/when video decoding is suspended due to operational errors, the video decoder may generate simulated video data by analyzing audio data received during the suspension period and developing video data from the data model and deviation(s) associated with patterns detected from the audio data.
    Type: Grant
    Filed: March 28, 2018
    Date of Patent: May 12, 2020
    Assignee: APPLE INC.
    Inventors: Xing Wen, Dazhong Zhang, Peikang Song, Xiaosong Zhou, Sudeng Hu, Hsi-Jung Wu, Jae Hoon Kim
  • Patent number: 10638147
    Abstract: Techniques are disclosed for managing reference frames for gradual coder refresh (GDR) operation. A GDR frame may be partitioned into a plurality of units, at least one of which is coded by instantaneous decoder refresh (IDR) techniques and other(s) of which are coded by other techniques such as inter-coding. The coded GDR frame may be exchanged between an encoder and a decoder. The encoder and decoder both may decode the GDR frame. The encoder and decoder may store the IDR-coded portion of the GDR frame in a reference picture buffer in a modified frame that includes, for the other portion(s) of the GDR frame, replacement content instead of the content obtained by decoding. The modified reference frame are expected by bias prediction search operations performed on later frame toward selection of the IDR-coded content as opposed to the replacement content.
    Type: Grant
    Filed: June 1, 2018
    Date of Patent: April 28, 2020
    Assignee: APPLE INC.
    Inventors: Sudeng Hu, Dazhong Zhang, Xing Wen, Peikang Song, Jae Hoon Kim, Hang Yuan, Xiaosong Zhou, Hsi-Jung Wu, Jingteng Xue
  • Publication number: 20190373270
    Abstract: Systems and methods for coding a video to be overlaid by annotations are devised. A motion compensated predictive coding is employed, wherein coding parameters of video pixel blocks are determined based on the pixel blocks' relation to the annotations. A decoder decodes the video and annotates it based on metadata, obtained from the coder or other sources, describing the annotations' appearance and rendering mode.
    Type: Application
    Filed: May 23, 2019
    Publication date: December 5, 2019
    Inventors: Sudeng HU, Xing WEN, Jae Hoon KIM, Peikang SONG, Hang YUAN, Dazhong ZHANG, Xiaosong ZHOU, Hsi-Jung WU, Christopher GARRIDO, Ming JIN, Patrick MIAUTON, Karthick SANTHANAM
  • Publication number: 20190373276
    Abstract: Techniques are disclosed for managing reference frames for gradual coder refresh (GDR) operation. A GDR frame may be partitioned into a plurality of units, at least one of which is coded by instantaneous decoder refresh (IDR) techniques and other(s) of which are coded by other techniques such as inter-coding. The coded GDR frame may be exchanged between an encoder and a decoder. The encoder and decoder both may decode the GDR frame. The encoder and decoder may store the IDR-coded portion of the GDR frame in a reference picture buffer in a modified frame that includes, for the other portion(s) of the GDR frame, replacement content instead of the content obtained by decoding. The modified reference frame are expected by bias prediction search operations performed on later frame toward selection of the IDR-coded content as opposed to the replacement content.
    Type: Application
    Filed: June 1, 2018
    Publication date: December 5, 2019
    Inventors: Sudeng Hu, Dazhong Zhang, Xing Wen, Peikang Song, Jae Hoon Kim, Hang Yuan, Xiaosong Zhou, Hsi-Jung Wu, Jingteng Xue
  • Publication number: 20190342351
    Abstract: Techniques presented herein provide an improved relay user experience and improved management of scarce computing and network resources as the number of relay endpoints increases. A sourcing endpoint device may generate a media feed, such as video and/or audio feed, representing contribution from a conference participant. The sourcing endpoint device may generate a priority value for the media feed, and the priority value may be transmitted to other members of the relay along with the input feed. Priority values of the different relay participants may be used by other devices, for example, intermediate servers or receiving endpoint devices, to manage aspects of the relay. For example, a relay server may prune streams from select endpoint devices based on relative priority values received from those devices. Alternatively, receiving endpoint devices may alter presentation of received feeds based on their associated priority values.
    Type: Application
    Filed: May 7, 2019
    Publication date: November 7, 2019
    Inventors: Christopher M. Garrido, Dazhong Zhang, Karthick Santhanam, Patrick Miauton, Xiaoxiao Zheng, Bess Chan, Peter Shiang, Sudeng Hu, Peikang Song, Xiaosong Zhou
  • Publication number: 20190306524
    Abstract: Techniques are disclosed for coding and decoding video data using object recognition and object modeling as a basis of coding and error recovery. A video decoder may decode coded video data received from a channel. The video decoder may perform object recognition on decoded video data obtained therefrom, and, when an object is recognized in the decoded video data, the video decoder may generate a model representing the recognized object. It may store data representing the model locally. The video decoder may communicate the model data to an encoder, which may form a basis of error mitigation and recovery. The video decoder also may monitor deviation patterns in the object model and associated patterns in audio content; if/when video decoding is suspended due to operational errors, the video decoder may generate simulated video data by analyzing audio data received during the suspension period and developing video data from the data model and deviation(s) associated with patterns detected from the audio data.
    Type: Application
    Filed: March 28, 2018
    Publication date: October 3, 2019
    Inventors: Xing Wen, Dazhong Zhang, Peikang Song, Xiaosong Zhou, Sudeng Hu, Hsi-Jung Wu, Jae Hoon Kim
  • Publication number: 20190014332
    Abstract: Techniques for encoding and decoding video images based on image content types are described. Techniques include determining a plurality of image content types from metadata or an image content type recognition algorithm, where each image content type corresponding to a portion of a source video, such as a spatial or temporal portion. Encoding parameters, such as quantization parameter, may be selected for portions of source by a constrained search for encoding parameters, where the constraints are based on image content type.
    Type: Application
    Filed: July 7, 2017
    Publication date: January 10, 2019
    Inventors: Peikang Song, Xing Wen, Sudeng Hu, Hang Yuan, Jae Hoon Kim, Dazhong Zhang, Xiaosong Zhou, Hsi-Jung Wu
  • Publication number: 20170359575
    Abstract: A video coder defines multiple fidelity regions in different spatial areas of a video sequence, each of which may have different fidelity characteristics. The coder may code the different representations in a common video sequence. Where prediction data crosses boundaries between the regions, interpolation may be performed to create like kind representations between prediction data and video content being coded.
    Type: Application
    Filed: June 5, 2017
    Publication date: December 14, 2017
    Inventors: Dazhong Zhang, Hang Yuan, Peikang Song, Jae Hoon Kim, Xing Wen, Sudeng Hu, Xiaosong Zhou, Chris Chung, Hsi-Jung Wu
  • Publication number: 20170359590
    Abstract: Techniques are disclosed for managing memory allocations when coding video data according to multiple codec configurations. According to these techniques, devices may negotiate parameters of a coding session that include parameters of a plurality of different codec configurations that may be used during the coding session. A device may estimate sizes of decoded picture buffers for each of the negotiated codec configurations and allocate in its memory a portion of memory sized according to a largest size of the estimated decoded picture buffers. Thereafter, the devices may exchange coded video data. The exchange may involve decoding coded data of reference pictures and storing the decoded reference pictures in the allocated memory. During the coding session, the devices may toggle among the different negotiated codec configurations. As they do, reallocations of memory may be avoided.
    Type: Application
    Filed: May 3, 2017
    Publication date: December 14, 2017
    Inventors: Dazhong Zhang, Hang Yuan, Peikang Song, Jae Hoon Kim, Xing Wen, Sudeng Hu, Xiaosong Zhou, Chris Y. Chung, Hsi-Jung Wu
  • Publication number: 20170359596
    Abstract: Video coding techniques are disclosed that can accommodate low bandwidth events and preserve visual quality, at least in areas of an image that have high significance to a viewer. Region(s) of interest may be identified from content of input frame that will be coded. Two representations of the input frame may be generated at different resolutions. A low resolution representation of the input frame may be coded according to predictive coding techniques in which a portion outside the region of interest is coded at higher quality than a portion inside the region of interest. A high resolution representation of the input frame may be coded according to predictive coding techniques in which a portion inside the region of interest is coded at higher quality than a portion outside the region of interest. Doing so preserves visual quality, at least in areas of the input image that correspond to the region of interest.
    Type: Application
    Filed: June 9, 2016
    Publication date: December 14, 2017
    Inventors: Jae Hoon Kim, Xiaosong ZHOU, Sudeng Hu, Chris CHUNG, Dazhong ZHANG, Hsi-Jung WU
  • Publication number: 20140307780
    Abstract: A bitstream corresponding to an encoded video is decoded. The encoded video includes a sequence of frames, and each frame is partitioned into encoded blocks. For each encoded block, an edge mode index is decoded based on an edge mode codeword and a prediction mode. The edge mode index indicates a subset of predetermined partitions selected from a partition library according to the prediction mode. The encoded block is partitioned based on the edge mode index to produce two or more block partitions. To each block partition, a coefficient rearrangement, an inverse transform and an inverse quantization is applied to produce a processed block partition. The processed block partitions are then combined into a decoded block for a video.
    Type: Application
    Filed: April 11, 2013
    Publication date: October 16, 2014
    Applicant: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Robert A. Cohen, Sudeng Hu, Anthony Vetro