Abstract: Systems, devices, methods, media, and instructions for automated image processing and content curation are described. In one embodiment a server computer system communicates at least a portion of a first content collection to a first client device, and receives a first selection communication in response, the first selection communication identifying a first piece of content of the first plurality of pieces of content. The server analyzes analyzing the first piece of content to identify a set of context values for the first piece of content, and accesses accessing a second content collection comprising pieces of content sharing at least a portion of the set of context values of the first piece of content. In various embodiments, different content values, image processing operations, and content selection operations are used to curate the content collections.

Abstract: A dense captioning system and method is provided for analyzing an image to generate proposed bounding regions for a plurality of visual concepts within the image, generating a region feature for each proposed bounding region to generate a plurality of region features of the image, and determining a context feature for the image using a proposed bounding region that is a largest in size of the proposed bounding regions. For each region feature of the plurality of region features of the image, the dense captioning system and method further provides for analyzing the region feature to determine for the region feature a detection score that indicates a likelihood that the region feature comprises an actual object, and generating a caption for a visual concept in the image using the region feature and the context feature when a detection score is above a specified threshold value.

Abstract: Systems, devices, media, and methods are presented for object detection and inserting graphical elements into an image stream in response to detecting the object. The systems and methods detect an object of interest in received frames of a video stream. The systems and methods identify a bounding box for the object of interest and estimate a three-dimensional position of the object of interest based on a scale of the object of interest. The systems and methods generate one or more graphical elements having a size based on the scale of the object of interest and a position based on the three-dimensional position estimated for the object of interest. The one or more graphical elements are generated within the video stream to form a modified video stream. The systems and methods cause presentation of the modified video stream including the object of interest and the one or more graphical elements.

Abstract: Systems and methods are provided for receiving, at a server computer, a plurality of content messages from a plurality of content sources, each content message comprising media content, for each of the plurality of content messages received, associating the media content with a predetermined media collection, and storing the content message in a database. The system and methods further providing for causing the plurality of content messages to be displayed on an operator device with other content messages associated with the media collection, determining that a predetermined trigger related to the media collection has been activated, updating an identifier of the media collection from a first indicator to a second indicator indicating an action needs to be taken on the media collection, and causing the updated identifier with the second indicator to be displayed on an operator device.

Abstract: Systems and methods are described for determining a first media item related to an event, of a plurality of stored media items each comprising video content related to the event, that was captured in a device orientation corresponding to a first device orientation detected for the first computing device; providing, to the first computing device, the first media item to be displayed on the first computing device; in response to a detected change to a second device orientation for the first computing device, determining a second media item that was captured in a device orientation corresponding to the second device orientation detected for the first computing device; and providing, to the first computing device, the second media item to be displayed on the first computing device.

Abstract: Systems and methods are described for determining that the user interaction with a display of a computing device during display of a video comprising a sequence of frames indicates a region of interest in a current frame of the sequence of frames of the displayed video. For each frame of the sequence of frames after the current frame, the frame is cropped to generate a cropped frame comprising a portion of the frame including the region of interest in the frame, the cropped frame is enlarged based on a display size corresponding to an angle or orientation of the computing device during display of the video, and the enlarged cropped frame replaces the frame such that the enlarged cropped frame is displayed in the sequence of frames of the video on the display of the computing device instead of the frame.

Abstract: Techniques are disclosed for image feature representation. The techniques exhibit discriminative power that can be used in any number of classification tasks, and are particularly effective with respect to fine-grained image classification tasks. In an embodiment, a given image to be classified is divided into image patches. A vector is generated for each image patch. Each image patch vector is compared to the Gaussian mixture components (each mixture component is also a vector) of a Gaussian Mixture Model (GMM). Each such comparison generates a similarity score for each image patch vector. For each Gaussian mixture component, the image patch vectors associated with a similarity score that is too low are eliminated. The selectively pooled vectors from all the Gaussian mixture components are then concatenated to form the final image feature vector, which can be provided to a classifier so the given input image can be properly categorized.

Abstract: Systems and methods are described for receiving, at a computing device, a plurality of video sources related to an event. For each video source of the plurality of video sources, the systems and methods further provide for analyzing, by the computing device, the video source of the plurality of video sources to determine a device orientation to associate with the video source, associating, by the computing device, the device orientation with the video source, and storing, by the computing device, the video source and the associated device orientation.

Abstract: Systems and methods are described for receiving, at a computing device, a video comprising a plurality of frames and determining, by the computing device, that vertical cropping should be performed for the video. For each frame of the plurality of frames, the computing device processes the video by analyzing the frame to determine a region of interest in the frame, wherein the frame is a first frame, cropping the first frame based on the region of interest in the frame to produce a vertically cropped frame for the video, determining a second frame immediately preceding the first frame, and smoothing a trajectory from the second frame to the vertically cropped frame. The vertically cropped frame is displayed to a user instead of the first frame.

Abstract: The present disclosure is directed to collaborative feature learning using social media data. For example, a machine learning system may identify social media data that includes user behavioral data, which indicates user interactions with content item. Using the identified social user behavioral data, the machine learning system may determine latent representations from the content items. In some embodiments, the machine learning system may train a machine-learning model based on the latent representations. Further, the machine learning system may extract features of the content item from the trained machine-learning model.

Abstract: A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.

Abstract: A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.

Abstract: Systems, devices, media, and methods are presented for identifying and categorically labeling objects within a set of images. The systems and methods receive an image depicting an object of interest, detect at least a portion of the object of interest within the image using a multilayer object model, determine context information, and identify the object of interest included in two or more bounding boxes.

Abstract: A machine learning system can generate an image mask (e.g., a pixel mask) comprising pixel assignments for pixels. The pixels can be assigned to classes, including, for example, face, clothes, body skin, or hair. The machine learning system can be implemented using a convolutional neural network that is configured to execute efficiently on computing devices having limited resources, such as mobile phones. The pixel mask can be used to more accurately display video effects interacting with a user or subject depicted in the image.

Abstract: Systems and methods are provided for receiving, at a first computing device, a request from a user to activate a new media collection, sending, by the first computing device, the request to a server computer for activation of the new media collection, receiving, by the first computing device, confirmation that the new media collection was activated, receiving, at the first computing device, a plurality of content messages associated with the new media collection, receiving, at the first computing device, from the user, a selection of the plurality of content messages to be included in the new media collection, sending, to the server computer, an indication of the selection of the content messages to be included in the new media collection, wherein the server computer causes the selection of content messages to be included in the new media collection and displayed in response to a request from at least a second computing device to view the new media collection.

Abstract: Systems, devices, media, and methods are presented for modeling facial representations using image segmentation with a client device. The systems and methods receive an image depicting a face, detect at least a portion of the face within the image, and identify a set of facial features within the portion of the face. The systems and methods generate a descriptor function representing the set of facial features, fit object functions of the descriptor function, identify an identification probability for each facial feature, and assign an identification to each facial feature.

Abstract: Systems, devices, methods, media, and instructions for automated image processing and content curation are described. In one embodiment a server computer system receives a plurality of content communications from a plurality of client devices, each content communication comprising an associated piece of content and a corresponding metadata. Each content communication is processed to determine associated context values for each piece of content, each associated context value comprising at least one content value generated by machine vision processing of the associated piece of content. A first content collection is automatically generated based on context values, and a set of user accounts are associated with the collection. An identifier associated with the first content collection is published to user devices associated with user accounts. In various additional embodiments, different content values, image processing operations, and content selection operations are used to curate content collections.

Abstract: Systems, devices, methods, media, and instructions for automated image processing and content curation are described. In one embodiment a server computer system receives a content message from a first content source, and analyzes the content message to determine one or more quality scores and one or more content values associated with the content message. The server computer system analyzes the content message with a plurality of content collections of the database to identify a match between at least one of the one or more content values and a topic associated with at least a first content collection of the one or more content collections and automatically adds the content message to the first content collection based at least in part on the match. In various embodiments, different content values, image processing operations, and content selection operations are used to curate content collections.

Abstract: Systems, devices, media, and methods are presented for identifying and categorically labeling objects within a set of images. The systems and methods receive an image depicting an object of interest, detect at least a portion of the object of interest within the image using a multilayer object model, determine context information, and identify the object of interest included in two or more bounding boxes.

Abstract: Systems and methods are provided for receiving, at a first computing device, a request from a user to activate a new media collection, sending the request to a server computer for activation of the new media collection, receiving confirmation that the new media collection was activated, receiving a plurality of content messages associated with the new media collection, receiving from the user, a selection of the plurality of content messages to be included in the new media collection, sending, to the server computer, an indication of the selection of the content messages to be included in the new media collection, wherein the server computer causes the selection of content messages to be included in the new media collection and displayed in response to a request from at least a second computing device to view the new media collection.