Abstract: Systems, methods, and software are described herein for removing people distractors from images. A distractor mitigation solution implemented in one or more computing devices detects people in an image and identifies salient regions in the image. The solution then determines a saliency cue for each person and classifies each person as wanted or as an unwanted distractor based at least on the saliency cue. An unwanted person is then removed from the image or otherwise reduced from the perspective of being an unwanted distraction.

Abstract: In implementations of object boundary generation, a computing device implements a boundary system to receive a mask defining a contour of an object depicted in a digital image, the mask having a lower resolution than the digital image. The boundary system maps a curve to the contour of the object and extracts strips of pixels from the digital image which are normal to points of the curve. A sample of the digital image is generated using the extracted strips of pixels which is input to a machine learning model. The machine learning model outputs a representation of a boundary of the object by processing the sample of the digital image.

Abstract: Fill techniques as implemented by a computing device are described to perform hole filling of a digital image. In one example, deeply learned features of a digital image using machine learning are used by a computing device as a basis to search a digital image repository to locate the guidance digital image. Once located, machine learning techniques are then used to align the guidance digital image with the hole to be filled in the digital image. Once aligned, the guidance digital image is then used to guide generation of fill for the hole in the digital image. Machine learning techniques are used to determine which parts of the guidance digital image are to be blended to fill the hole in the digital image and which parts of the hole are to receive new content that is synthesized by the computing device.

Abstract: Methods and systems are provided for optimal segmentation of an image based on multiple segmentations. In particular, multiple segmentation methods can be combined by taking into account previous segmentations. For instance, an optimal segmentation can be generated by iteratively integrating a previous segmentation (e.g., using an image segmentation method) with a current segmentation (e.g., using the same or different image segmentation method). To allow for optimal segmentation of an image based on multiple segmentations, one or more neural networks can be used. For instance, a convolutional RNN can be used to maintain information related to one or more previous segmentations when transitioning from one segmentation method to the next. The convolutional RNN can combine the previous segmentation(s) with the current segmentation without requiring any information about the image segmentation method(s) used to generate the segmentations.

Abstract: A discriminative captioning system generates captions for digital images that can be used to tell two digital images apart. The discriminative captioning system includes a machine learning system that is trained by a discriminative captioning training system that includes a retrieval machine learning system. For training, a digital image is input to the caption generation machine learning system, which generates a caption for the digital image. The digital image and the generated caption, as well as a set of additional images, are input to the retrieval machine learning system. The retrieval machine learning system generates a discriminability loss that indicates how well the retrieval machine learning system is able to use the caption to discriminate between the digital image and each image in the set of additional digital images. This discriminability loss is used to train the caption generation machine learning system.

Abstract: Semantic segmentation techniques and systems are described that overcome the challenges of limited availability of training data to describe the potentially millions of tags that may be used to describe semantic classes in digital images. In one example, the techniques are configured to train neural networks to leverage different types of training datasets using sequential neural networks and use of vector representations to represent the different semantic classes.

Abstract: Fill techniques as implemented by a computing device are described to perform hole filling of a digital image. In one example, deeply learned features of a digital image using machine learning are used by a computing device as a basis to search a digital image repository to locate the guidance digital image. Once located, machine learning techniques are then used to align the guidance digital image with the hole to be filled in the digital image. Once aligned, the guidance digital image is then used to guide generation of fill for the hole in the digital image. Machine learning techniques are used to determine which parts of the guidance digital image are to be blended to fill the hole in the digital image and which parts of the hole are to receive new content that is synthesized by the computing device.

Abstract: In implementations of object boundary generation, a computing device implements a boundary system to receive a mask defining a contour of an object depicted in a digital image, the mask having a lower resolution than the digital image. The boundary system maps a curve to the contour of the object and extracts strips of pixels from the digital image which are normal to points of the curve. A sample of the digital image is generated using the extracted strips of pixels which is input to a machine learning model. The machine learning model outputs a representation of a boundary of the object by processing the sample of the digital image.

Abstract: Systems, methods, and software are described herein for removing people distractors from images. A distractor mitigation solution implemented in one or more computing devices detects people in an image and identifies salient regions in the image. The solution then determines a saliency cue for each person and classifies each person as wanted or as an unwanted distractor based at least on the saliency cue. An unwanted person is then removed from the image or otherwise reduced from the perspective of being an unwanted distraction.

Abstract: Fill techniques as implemented by a computing device are described to perform hole filling of a digital image. In one example, deeply learned features of a digital image using machine learning are used by a computing device as a basis to search a digital image repository to locate the guidance digital image. Once located, machine learning techniques are then used to align the guidance digital image with the hole to be filled in the digital image. Once aligned, the guidance digital image is then used to guide generation of fill for the hole in the digital image. Machine learning techniques are used to determine which parts of the guidance digital image are to be blended to fill the hole in the digital image and which parts of the hole are to receive new content that is synthesized by the computing device.

Abstract: In implementations of object boundary generation, a computing device implements a boundary system to receive a mask defining a contour of an object depicted in a digital image, the mask having a lower resolution than the digital image. The boundary system maps a curve to the contour of the object and extracts strips of pixels from the digital image which are normal to points of the curve. A sample of the digital image is generated using the extracted strips of pixels which is input to a machine learning model. The machine learning model outputs a representation of a boundary of the object by processing the sample of the digital image.

Abstract: Methods and systems are provided for optimal segmentation of an image based on multiple segmentations. In particular, multiple segmentation methods can be combined by taking into account previous segmentations. For instance, an optimal segmentation can be generated by iteratively integrating a previous segmentation (e.g., using an image segmentation method) with a current segmentation (e.g., using the same or different image segmentation method). To allow for optimal segmentation of an image based on multiple segmentations, one or more neural networks can be used. For instance, a convolutional RNN can be used to maintain information related to one or more previous segmentations when transitioning from one segmentation method to the next. The convolutional RNN can combine the previous segmentation(s) with the current segmentation without requiring any information about the image segmentation method(s) used to generate the segmentations.

Abstract: In implementations of object boundary generation, a computing device implements a boundary system to receive a mask defining a contour of an object depicted in a digital image, the mask having a lower resolution than the digital image. The boundary system maps a curve to the contour of the object and extracts strips of pixels from the digital image which are normal to points of the curve. A sample of the digital image is generated using the extracted strips of pixels which is input to a machine learning model. The machine learning model outputs a representation of a boundary of the object by processing the sample of the digital image.

Abstract: A table layout determination system implemented on a computing device obtains an image of a table having multiple cells. The table layout determination system includes a row prediction machine learning system that generates, for each of multiple rows of pixels in the image of the table, a probability of the row being a row separator, and a column prediction machine learning system generates, for each of multiple columns of pixels in the image of the table, a probability of the column being a column separator. An inference system uses these probabilities of the rows being row separators and the columns being column separators to identify the row separators and column separators for the table. These row separators and column separators are the layout of the table.

Abstract: Fill techniques as implemented by a computing device are described to perform hole filling of a digital image. In one example, deeply learned features of a digital image using machine learning are used by a computing device as a basis to search a digital image repository to locate the guidance digital image. Once located, machine learning techniques are then used to align the guidance digital image with the hole to be filled in the digital image. Once aligned, the guidance digital image is then used to guide generation of fill for the hole in the digital image. Machine learning techniques are used to determine which parts of the guidance digital image are to be blended to fill the hole in the digital image and which parts of the hole are to receive new content that is synthesized by the computing device.

Abstract: Fill techniques as implemented by a computing device are described to perform hole filling of a digital image. In one example, deeply learned features of a digital image using machine learning are used by a computing device as a basis to search a digital image repository to locate the guidance digital image. Once located, machine learning techniques are then used to align the guidance digital image with the hole to be filled in the digital image. Once aligned, the guidance digital image is then used to guide generation of fill for the hole in the digital image. Machine learning techniques are used to determine which parts of the guidance digital image are to be blended to fill the hole in the digital image and which parts of the hole are to receive new content that is synthesized by the computing device.

Abstract: A table layout determination system implemented on a computing device obtains an image of a table having multiple cells. The table layout determination system includes a row prediction machine learning system that generates, for each of multiple rows of pixels in the image of the table, a probability of the row being a row separator, and a column prediction machine learning system generates, for each of multiple columns of pixels in the image of the table, a probability of the column being a column separator. An inference system uses these probabilities of the rows being row separators and the columns being column separators to identify the row separators and column separators for the table. These row separators and column separators are the layout of the table.

Abstract: Systems and techniques are described herein for directing a user conversation to obtain an editing query, and removing and replacing objects in an image based on the editing query. Pixels corresponding to an object in the image indicated by the editing query are ascertained. The editing query is processed to determine whether it includes a remove request or a replace request. A search query is constructed to obtain images, such as from a database of stock images, including fill material or replacement material to fulfill the remove request or replace request, respectively. Composite images are generated from the fill material or the replacement material and the image to be edited. Composite images are harmonized to remove editing artifacts and make the images look natural. A user interface exposes images, and the user interface accepts multi-modal user input during the directed user conversation.

Abstract: A discriminative captioning system generates captions for digital images that can be used to tell two digital images apart. The discriminative captioning system includes a machine learning system that is trained by a discriminative captioning training system that includes a retrieval machine learning system. For training, a digital image is input to the caption generation machine learning system, which generates a caption for the digital image. The digital image and the generated caption, as well as a set of additional images, are input to the retrieval machine learning system. The retrieval machine learning system generates a discriminability loss that indicates how well the retrieval machine learning system is able to use the caption to discriminate between the digital image and each image in the set of additional digital images. This discriminability loss is used to train the caption generation machine learning system.

Abstract: Semantic segmentation techniques and systems are described that overcome the challenges of limited availability of training data to describe the potentially millions of tags that may be used to describe semantic classes in digital images. In one example, the techniques are configured to train neural networks to leverage different types of training datasets using sequential neural networks and use of vector representations to represent the different semantic classes.