Abstract: A machine learning model trained with a triplet loss function classifies input strings into one of multiple hierarchical categories. The machine learning model is pre-trained using masking language modeling on a corpus of unlabeled strings. The machine learning module includes an attention-based bi-directional transformer layer. Following initial training, the machine learning model is refined by additional training with a loss function that includes cross-entropy loss and triplet loss. This provides a deep learning solution to classify input strings into one or more hierarchical categories. Embeddings generated from inputs to the machine learning model capture language similarities that can be visualized in a cartesian plane where strings with similar meanings are grouped together.

Abstract: The present disclosure relates to selectively analyzing digital content items from a social networking system to generate a searchable graph object that facilitates visualization of correlations between entities of interest (e.g., brands, products, services, companies), concepts (e.g., domain-specific and/or general terms), hashtags, and other output classes. For example, systems described herein involve collecting images and/or videos that are publicly available via a social networking platform and evaluating content and metadata thereof to identify entities of interest therein and predict various concepts therein to generate a graph object that provides a searchable matrix. The systems herein provide a mechanism for processing a graph query that presents a relational graph showing correlations between the query and the various output classes.

Abstract: A method for enhancing tangible content on a physical activity surface is described. In an example implementation, the method includes capturing, using a video capture device of a computing device, a video stream that includes an activity scene of a physical activity surface; detecting in the video stream, using a detector executable on the computing device, a tangible content item on the physical activity surface; recognizing, from the video stream, one or more visually instructive elements in the tangible content item; determining a tangible identifier based on the one or more visually instructive elements in the tangible content item; retrieving a digital content item using the tangible identifier; and providing the digital content item on the computing device.

Abstract: The present disclosure relates to generating a complex entity index based on a combination of atomic and deep learned attributes associated with instances of a complex entity. For example, systems described herein generate a multi-dimensional representation of entity instances based on evaluation of digital content associated with the respective entity instances. Systems described herein further generate an index representation in which similarity of entity instances are illustrated and presented via an interactive presentation that enables a user to traverse instances of an entity to observe similarities and differences between instances of an entity that have similar embeddings to one another within a multi-dimensional index space.

Abstract: A data processing system implements obtaining query parameters for a query for content items in a datastore, the query parameters including attributes of content items for which a search is to be conducted; obtaining a first set of content items from a content datastore based on the query parameters; analyzing the first set of content items using a first machine learning model trained to generate relevant content information that identifies a plurality of relevant content items included in the first set of content items; and analyzing the plurality of relevant content items using a second machine learning model configured to output novel content information, the novel content information including a plurality of content items predicted to be relevant and novel, the novel content information ranking the plurality of content items predicted to be relevant and novel based on a novelty score associated with each respective content item.

Abstract: An example system includes a stand configured to position a computing device proximate to a physical activity surface. The system further includes a video capture device, a detector, and an activity application. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically interactable with by a user. The detector is executable to detect motion in the activity scene based on the processing and, responsive to detecting the motion, process the video stream to detect one or more interface objects included in the activity scene of the physical activity surface. The activity application is executable to present virtual information on a display of the computing device based on the one or more detected interface objects.

Abstract: A data processing system implements obtaining query parameters for a query for content items in a datastore, the query parameters including attributes of content items for which a search is to be conducted; obtaining a first set of content items from a content datastore based on the query parameters; analyzing the first set of content items using a first machine learning model trained to generate relevant content information that identifies a plurality of relevant content items included in the first set of content items; and analyzing the plurality of relevant content items using a second machine learning model configured to output novel content information, the novel content information including a plurality of content items predicted to be relevant and novel, the novel content information ranking the plurality of content items predicted to be relevant and novel based on a novelty score associated with each respective content item.

Abstract: An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.

Abstract: A computer implemented method includes receiving an image that includes a type of object, segmenting the object into multiple segments via a trained segmentation machine learning model, and inputting the segments into multiple different attribute extraction models to extract different types of attributes from each of the multiple segments.

Abstract: The present disclosure relates to selectively analyzing digital content items from a social networking system to generate a searchable graph object that facilitates visualization of correlations between entities of interest (e.g., brands, products, services, companies), concepts (e.g., domain-specific and/or general terms), hashtags, and other output classes. For example, systems described herein involve collecting images and/or videos that are publicly available via a social networking platform and evaluating content and metadata thereof to identify entities of interest therein and predict various concepts therein to generate a graph object that provides a searchable matrix. The systems herein provide a mechanism for processing a graph query that presents a relational graph showing correlations between the query and the various output classes.

Abstract: Systems and methods are provided for building and training machine learning models configured to generate in-domain embeddings and perform multimodal analysis inside the same domain. The models include a first encoder trained to receive input from one or more entities represented in a first modality and to encode the one or more entities in the first modality, such that the first encoder is configured to output a first set of embeddings. The models also include a second encoder trained to receive input from one or more entities represented in the second modality and to encode the one or more entities in the second modality, such that the second encoder is configured to output a second set of embeddings. The models also include a projection layer configured to project the first set of embeddings and the second set of embeddings to a shared contrastive space.

Abstract: The present disclosure relates to generating a complex entity index based on a combination of atomic and deep learned attributes associated with instances of a complex entity. For example, systems described herein generate a multi-dimensional representation of entity instances based on evaluation of digital content associated with the respective entity instances. Systems described herein further generate an index representation in which similarity of entity instances are illustrated and presented via an interactive presentation that enables a user to traverse instances of an entity to observe similarities and differences between instances of an entity that have similar embeddings to one another within a multi-dimensional index space.

Abstract: An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.

Abstract: A computer implemented method includes rendering a document page as an image; detecting tables, columns, and other associated table objects within the image via one or more table recognition models that model objects in the image as overlapping bounding boxes; transforming the set of objects into a structured representation of the table; extracting data from the objects into the structured representation; and exporting the table into the desired output format.

Abstract: A method and system for organizing a virtual classroom session. In an example implementation, a method includes, receiving first media data including a first user media stream depicting a first user and a first workspace media stream depicting a first physical activity scene that is proximate a first computing device of the first user and a second media data including a second user media stream depicting a second user and a second workspace media stream depicting a second physical activity scene that is proximate a second computing device of the second user, generating a graphical virtual meeting user interface and providing the graphical virtual meeting user interface for display.

Abstract: An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.

Abstract: Activity scene detection, display, and enhancement implementations are described. In an example implementation, a method includes displaying an animated character on a display of a computing device, detecting a tangible interface object on a physical activity scene proximate to the computing device, rendering a virtual interface object based on the tangible interface object, determining an interaction routine between the animated character and the virtual interface object, and executing the animation routine to animate on the display, an interaction between the animated character and the virtual interface object.

Abstract: An example system includes a stand configured to position a computing device proximate to a physical activity surface. The system further includes a video capture device, a detector, and an activity application. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically interactable with by a user. The detector is executable to detect motion in the activity scene based on the processing and, responsive to detecting the motion, process the video stream to detect one or more interface objects included in the activity scene of the physical activity surface. The activity application is executable to present virtual information on a display of the computing device based on the one or more detected interface objects.

Abstract: Activity scene detection, display, and enhancement implementations are described. In an example implementation, a method includes displaying an animated character on a display of a computing device, detecting a tangible interface object on a physical activity scene proximate to the computing device, rendering a virtual interface object based on the tangible interface object, determining an interaction routine between the animated character and the virtual interface object, and executing the animation routine to animate on the display, an interaction between the animated character and the virtual interface object.

Abstract: An example system includes a stand configured to position a computing device proximate to a physical activity surface. The system further includes a video capture device, a detector, and an activity application. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically interactable with by a user. The detector is executable to detect motion in the activity scene based on the processing and, responsive to detecting the motion, process the video stream to detect one or more interface objects included in the activity scene of the physical activity surface. The activity application is executable to present virtual information on a display of the computing device based on the one or more detected interface objects.