Patents by Inventor Zhe Lin

Zhe Lin has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20210365727
    Abstract: Text-to-visual machine learning embedding techniques are described that overcome the challenges of conventional techniques in a variety of ways. These techniques include use of query-based training data which may expand availability and types of training data usable to train a model. Generation of negative digital image samples is also described that may increase accuracy in training the model using machine learning. A loss function is also described that also supports increased accuracy and computational efficiency by losses separately, e.g., between positive or negative sample embeddings a text embedding.
    Type: Application
    Filed: August 10, 2021
    Publication date: November 25, 2021
    Applicant: Adobe Inc.
    Inventors: Pranav Vineet Aggarwal, Zhe Lin, Baldo Antonio Faieta, Saeid Motiian
  • Patent number: 11184558
    Abstract: Systems and methods provide reframing operations in a smart editing system that may generate a focal point within a mask of an object for each frame of a video segment and perform editing effects on the frames of the video segment to quickly provide users with natural video editing effects. A reframing engine may processes video clips using a segmentation and hotspot module to determine a salient region of an object, generate a mask of the object, and track the trajectory of an object in the video clips. The reframing engine may then receive reframing parameters from a crop suggestion module and a user interface. Based on the determined trajectory of an object in a video clip and reframing parameters, the reframing engine may use reframing logic to produce temporally consistent reframing effects relative to an object for the video clip.
    Type: Grant
    Filed: June 12, 2020
    Date of Patent: November 23, 2021
    Assignee: Adobe Inc.
    Inventors: Lu Zhang, Jianming Zhang, Zhe Lin, Radomir Mech
  • Publication number: 20210358130
    Abstract: The present disclosure relates to an object selection system that accurately detects and automatically selects user-requested objects (e.g., query objects) in a digital image. For example, the object selection system builds and utilizes an object selection pipeline to determine which object detection neural network to utilize to detect a query object based on analyzing the object class of the query object. In addition, the object selection system can add, update, or replace portions of the object selection pipeline to improve overall accuracy and efficiency of automatic object selection within an image.
    Type: Application
    Filed: July 28, 2021
    Publication date: November 18, 2021
    Inventors: Scott Cohen, Zhe Lin, Mingyang Ling
  • Publication number: 20210357684
    Abstract: A panoptic labeling system includes a modified panoptic labeling neural network (“modified PLNN”) that is trained to generate labels for pixels in an input image. The panoptic labeling system generates modified training images by combining training images with mask instances from annotated images. The modified PLNN determines a set of labels representing categories of objects depicted in the modified training images. The modified PLNN also determines a subset of the labels representing categories of objects depicted in the input image. For each mask pixel in a modified training image, the modified PLNN calculates a probability indicating whether the mask pixel has the same label as an object pixel. The modified PLNN generates a mask label for each mask pixel, based on the probability. The panoptic labeling system provides the mask label to, for example, a digital graphics editing system that uses the labels to complete an infill operation.
    Type: Application
    Filed: May 13, 2020
    Publication date: November 18, 2021
    Inventors: Sohrab Amirghodsi, Zhe Lin, Yilin Wang, Tianshu Yu, Connelly Barnes, Elya Shechtman
  • Publication number: 20210356633
    Abstract: Provided is a wide angle application high reflective mirror having a reflection band partially overlapping in a wavelength range of 800-4000 nm. The mirror comprises a film system in which a plurality of high refractive index film layers and a plurality of low refractive index film layers that are alternately stacked, and the material of the high refractive index film layer is one of SiH, SiOxHy, or SiOxNy, or a mixture thereof. The highly reflective mirror can achieve a reflectance greater than 99% with an incident angle ranging from 0 to 60 degrees over a large angle range.
    Type: Application
    Filed: December 9, 2018
    Publication date: November 18, 2021
    Inventors: Li WU, Zhe LIU, Guanglong YU, Yu LI, Yan SU, Zhiqiang LIN
  • Publication number: 20210350504
    Abstract: Methods and systems are provided for generating enhanced image. A neural network system is trained where the training includes training a first neural network that generates enhanced images conditioned on content of an image undergoing enhancement and training a second neural network that designates realism of the enhanced images generated by the first neural network. The neural network system is trained by determine loss and accordingly adjusting the appropriate neural network(s). The trained neural network system is used to generate an enhanced aesthetic image from a selected image where the output enhanced aesthetic image has increased aesthetics when compared to the selected image.
    Type: Application
    Filed: July 19, 2021
    Publication date: November 11, 2021
    Inventors: Xiaohui Shen, Zhe Lin, Xin Lu, Sarah Aye Kong, I-Ming Pao, Yingcong Chen
  • Publication number: 20210342983
    Abstract: Methods and systems are provided for accurately filling holes, regions, and/or portions of images using iterative image inpainting. In particular, iterative inpainting utilize a confidence analysis of predicted pixels determined during the iterations of inpainting. For instance, a confidence analysis can provide information that can be used as feedback to progressively fill undefined pixels that comprise the holes, regions, and/or portions of an image where information for those respective pixels is not known. To allow for accurate image inpainting, one or more neural networks can be used. For instance, a coarse result neural network (e.g., a GAN comprised of a generator and a discriminator) and a fine result neural network (e.g., a GAN comprised of a generator and two discriminators).
    Type: Application
    Filed: April 29, 2020
    Publication date: November 4, 2021
    Inventors: Zhe LIN, Yu ZENG, Jimei YANG, Jianming ZHANG, Elya SHECHTMAN
  • Publication number: 20210342984
    Abstract: Methods and systems are provided for accurately filling holes, regions, and/or portions of high-resolution images using guided upsampling during image inpainting. For instance, an image inpainting system can apply guided upsampling to an inpainted image result to enable generation of a high-resolution inpainting result from a lower-resolution image that has undergone inpainting. To allow for guided upsampling during image inpainting, one or more neural networks can be used. For instance, a low-resolution result neural network (e.g., comprised of an encoder and a decoder) and a high-resolution input neural network (e.g., comprised of an encoder and a decoder). The image inpainting system can use such networks to generate a high-resolution inpainting image result that fills the hole, region, and/or portion of the image.
    Type: Application
    Filed: May 1, 2020
    Publication date: November 4, 2021
    Inventors: Zhe LIN, Yu ZENG, Jimei YANG, Jianming ZHANG, Elya SHECHTMAN
  • Publication number: 20210336011
    Abstract: An epitaxial structure includes a substrate, a buffer layer, a channel layer, a barrier layer, a diffusion barrier layer, and a P-type gallium nitride layer sequentially stacked from bottom to top. The P-type gallium nitride layer has a first lattice constant. The diffusion barrier layer includes a chemical composition of Inx1Aly1Gaz1N, where x1+y1+z1=1, 0?x1?0.3, 0?y1?1.0, and 0?z1?1.0. The chemical composition of the diffusion barrier layer has a proportional relationship so that the diffusion barrier layer has a second lattice constant that matches the first lattice constant, and the second lattice constant is between 80% and 120% of the first lattice constant.
    Type: Application
    Filed: April 23, 2021
    Publication date: October 28, 2021
    Inventors: TZU-YAO LIN, JIA-ZHE LIU, YING-RU SHIH
  • Patent number: 11158055
    Abstract: The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.
    Type: Grant
    Filed: July 26, 2019
    Date of Patent: October 26, 2021
    Assignee: ADOBE INC.
    Inventors: Zhe Lin, Jianming Zhang, He Zhang, Federico Perazzi
  • Patent number: 11152032
    Abstract: The present disclosure is directed toward systems and methods for tracking objects in videos. For example, one or more embodiments described herein utilize various tracking methods in combination with an image search index made up of still video frames indexed from a video. One or more embodiments described herein utilize a backward and forward tracking method that is anchored by one or more key frames in order to accurately track an object through the frames of a video, even when the video is long and may include challenging conditions.
    Type: Grant
    Filed: April 25, 2019
    Date of Patent: October 19, 2021
    Assignee: ADOBE INC.
    Inventors: Zhihong Ding, Zhe Lin, Xiaohui Shen, Michael Kaplan, Jonathan Brandt
  • Publication number: 20210319255
    Abstract: The present disclosure relates to an object selection system that automatically detects and selects objects in a digital image utilizing a large-scale object detector. For instance, in response to receiving a request to automatically select a query object with an unknown object class in a digital image, the object selection system can utilize a large-scale object detector to detect potential objects in the image, filter out one or more potential objects, and label the remaining potential objects in the image to detect the query object. In some implementations, the large-scale object detector utilizes a region proposal model, a concept mask model, and an auto tagging model to automatically detect objects in the digital image.
    Type: Application
    Filed: May 26, 2021
    Publication date: October 14, 2021
    Inventors: Khoi Pham, Scott Cohen, Zhe Lin, Zhihong Ding, Walter Wei Tuh Chang
  • Publication number: 20210319056
    Abstract: The present disclosure relates to a retrieval method including: generating a graph representing a set of users, items, and queries; generating clusters from the media items; generating embeddings for each cluster from embeddings of the items within the corresponding cluster; generating augmented query embeddings for each cluster from the embedding of the corresponding cluster and query embeddings of the queries; inputting the cluster embeddings and the augmented query embeddings to a layer of a graph convolutional network (GCN) to determine user embeddings of the users; inputting the embedding of the given user and a query embedding of the given query to a layer of the GCN to determine a user-specific query embedding; generating a score for each of the items based on the item embeddings and the user-specific query embedding; and presenting the items having the score exceeding a threshold.
    Type: Application
    Filed: April 8, 2020
    Publication date: October 14, 2021
    Inventors: Handong Zhao, Ajinkya Kale, Xiaowei Jia, Zhe Lin
  • Publication number: 20210319232
    Abstract: A Video Semantic Segmentation System (VSSS) is disclosed that performs accurate and fast semantic segmentation of videos using a set of temporally distributed neural networks. The VSSS receives as input a video signal comprising a contiguous sequence of temporally-related video frames. The VSSS extracts features from the video frames in the contiguous sequence and based upon the extracted features, selects, from a set of labels, a label to be associated with each pixel of each video frame in the video signal. In certain embodiments, a set of multiple neural networks are used to extract the features to be used for video segmentation and the extraction of features is distributed among the multiple neural networks in the set. A strong feature representation representing the entirety of the features is produced for each video frame in the sequence of video frames by aggregating the output features extracted by the multiple neural networks.
    Type: Application
    Filed: April 13, 2020
    Publication date: October 14, 2021
    Inventors: Federico Perazzi, Zhe Lin, Ping Hu, Oliver Wang, Fabian David Caba Heilbron
  • Patent number: 11144784
    Abstract: Text-to-visual machine learning embedding techniques are described that overcome the challenges of conventional techniques in a variety of ways. These techniques include use of query-based training data which may expand availability and types of training data usable to train a model. Generation of negative digital image samples is also described that may increase accuracy in training the model using machine learning. A loss function is also described that also supports increased accuracy and computational efficiency by losses separately, e.g., between positive or negative sample embeddings a text embedding.
    Type: Grant
    Filed: May 30, 2019
    Date of Patent: October 12, 2021
    Assignee: Adobe Inc.
    Inventors: Pranav Vineet Aggarwal, Zhe Lin, Baldo Antonio Faieta, Saeid Motiian
  • Patent number: 11138257
    Abstract: Object search techniques for digital images are described. In the techniques described herein, semantic features are extracted on a per-object basis form a digital image. This supports location of objects within digital images and is not limited to semantic features of an entirety of the digital image as involved in conventional image similarity search techniques. This may be combined with indications a location of the object globally with respect to the digital image through use of a global segmentation mask, use of a local segmentation mask to capture post and characteristics of the object itself, and so on.
    Type: Grant
    Filed: January 16, 2020
    Date of Patent: October 5, 2021
    Assignee: Adobe Inc.
    Inventors: Midhun Harikumar, Zhe Lin, Pramod Srinivasan, Jianming Zhang, Daniel David Miranda, Baldo Antonio Faieta
  • Patent number: 11126890
    Abstract: Systems and methods are described for object detection within a digital image using a hierarchical softmax function. The method may include applying a first softmax function of a softmax hierarchy on a digital image based on a first set of object classes that are children of a root node of a class hierarchy, then apply a second (and subsequent) softmax functions to the digital image based on a second (and subsequent) set of object classes, where the second (and subsequent) object classes are children nodes of an object class from the first (or parent) object classes. The methods may then include generating an object recognition output using a convolutional neural network (CNN) based at least in part on applying the first and second (and subsequent) softmax functions. In some cases, the hierarchical softmax function is the loss function for the CNN.
    Type: Grant
    Filed: April 18, 2019
    Date of Patent: September 21, 2021
    Assignee: ADOBE INC.
    Inventors: Zhe Lin, Mingyang Ling, Jianming Zhang, Jason Kuen, Federico Perazzi, Brett Butterfield, Baldo Faieta
  • Patent number: 11127139
    Abstract: Enhanced methods and systems for the semantic segmentation of images are described. A refined segmentation mask for a specified object visually depicted in a source image is generated based on a coarse and/or raw segmentation mask. The refined segmentation mask is generated via a refinement process applied to the coarse segmentation mask. The refinement process correct at least a portion of both type I and type II errors, as well as refine boundaries of the specified object, associated with the coarse segmentation mask. Thus, the refined segmentation mask provides a more accurate segmentation of the object than the coarse segmentation mask. A segmentation refinement model is employed to generate the refined segmentation mask based on the coarse segmentation mask. That is, the segmentation model is employed to refine the coarse segmentation mask to generate more accurate segmentations of the object. The refinement process is an iterative refinement process carried out via a trained neural network.
    Type: Grant
    Filed: September 18, 2019
    Date of Patent: September 21, 2021
    Assignee: ADOBE INC.
    Inventors: Jianming Zhang, Zhe Lin
  • Publication number: 20210289010
    Abstract: A computer device receives a session message in a group session on a social networking application. The session message includes a child application identifier corresponding to a child application. The device determines a session identifier corresponding to the group session to which the session message belongs. The device transmits the session identifier to the child application and receives from the child application a request for page data corresponding to the session identifier. In accordance with the request, the device obtains the page data corresponding to the child application identifier. The device renders a child application page in a child application according to the obtained data. In accordance with detecting a trigger event corresponding to the child application identifier on the session message, the device invokes the child application in the social networking application.
    Type: Application
    Filed: June 1, 2021
    Publication date: September 16, 2021
    Inventors: Weibang LUO, Jisheng HUANG, Haojun HU, Qingjie LIN, Wenxiao ZHENG, Yi DUAN, Zhaowei WANG, Chao LIN, Canhui HUANG, Hao HU, Zhe CHENG, Yu WU, Xuyu GUI, Zhaopeng LIANG, Yifan ZHU, Yuyang PENG, Kunli DENG
  • Publication number: 20210271794
    Abstract: The present disclosure describes a method for optimizing metal cuts in standard cells. The method includes placing a standard cell in a layout area and inserting a metal cut along a metal interconnect of the standard cell at a location away from a boundary of the standard cell. The method further includes disconnecting, at the location, a metal portion of the metal interconnect from a remaining portion of the metal interconnect based on the metal cut.
    Type: Application
    Filed: May 17, 2021
    Publication date: September 2, 2021
    Applicant: Taiwan Semiconductor Manufacturing Co., Ltd.
    Inventors: Cheok-Kei Lei, Chi-Lin Liu, Hui-Zhong Zhuang, Zhe-Wei Jiang, Chi-Yu Lu, Yi-Hsin Ko