Abstract: A molecule representation is extracted from a document and associated with the document in a metadata database. For example, an image of a molecular structure may be extracted from a document and stored in the metadata database in a text-based representation such as SMILES. The metadata database may be searched to identify documents that mention a particular molecule. Continuing the example, the metadata database may be searched with a SMILES representation to identify the document and other documents that refer to the same molecule. The metadata database may index documents based on different types of molecule representations, including text-based, image-based, graph-based, name, abbreviation, etc. This allows search over multiple representations of a molecule, improving accuracy and thoroughness. These improvements reduce the time and computational resources needed to search for documents that refer to a particular molecule.

Abstract: Machine learning models are used to invoke a sequence of tools in response to a user request. A list of available tools is culled before a machine learning model selects from the remaining tools to generate a plan. The plan, which describes tool invocations in natural language, is then converted to code that can programmatically invoke the tools. In some configurations, the list of available tools is first culled by removing tools that do not appear in similarly described plans. The list may further be culled by removing tools that do not appear in plans generated by machine learning models. Then, a machine learning model is prompted to generate a final plan from the culled list and the user request. In some configurations, the final plan is converted to code by prompting a machine learning model to extrapolate from the code of similar plans.

Abstract: A computer implemented method includes rendering a document page as an image; detecting tables, columns, and other associated table objects within the image via one or more table recognition models that model objects in the image as overlapping bounding boxes; transforming the set of objects into a structured representation of the table; extracting data from the objects into the structured representation; and exporting the table into the desired output format.

Abstract: The present disclosure relates to selectively analyzing digital content items from a social networking system to generate a searchable graph object that facilitates visualization of correlations between entities of interest (e.g., brands, products, services, companies), concepts (e.g., domain-specific and/or general terms), hashtags, and other output classes. For example, systems described herein involve collecting images and/or videos that are publicly available via a social networking platform and evaluating content and metadata thereof to identify entities of interest therein and predict various concepts therein to generate a graph object that provides a searchable matrix. The systems herein provide a mechanism for processing a graph query that presents a relational graph showing correlations between the query and the various output classes.

Abstract: A machine learning model trained with a triplet loss function classifies input strings into one of multiple hierarchical categories. The machine learning model is pre-trained using masking language modeling on a corpus of unlabeled strings. The machine learning module includes an attention-based bi-directional transformer layer. Following initial training, the machine learning model is refined by additional training with a loss function that includes cross-entropy loss and triplet loss. This provides a deep learning solution to classify input strings into one or more hierarchical categories. Embeddings generated from inputs to the machine learning model capture language similarities that can be visualized in a cartesian plane where strings with similar meanings are grouped together.

Abstract: The present disclosure relates to selectively analyzing digital content items from a social networking system to generate a searchable graph object that facilitates visualization of correlations between entities of interest (e.g., brands, products, services, companies), concepts (e.g., domain-specific and/or general terms), hashtags, and other output classes. For example, systems described herein involve collecting images and/or videos that are publicly available via a social networking platform and evaluating content and metadata thereof to identify entities of interest therein and predict various concepts therein to generate a graph object that provides a searchable matrix. The systems herein provide a mechanism for processing a graph query that presents a relational graph showing correlations between the query and the various output classes.

Abstract: A method for enhancing tangible content on a physical activity surface is described. In an example implementation, the method includes capturing, using a video capture device of a computing device, a video stream that includes an activity scene of a physical activity surface; detecting in the video stream, using a detector executable on the computing device, a tangible content item on the physical activity surface; recognizing, from the video stream, one or more visually instructive elements in the tangible content item; determining a tangible identifier based on the one or more visually instructive elements in the tangible content item; retrieving a digital content item using the tangible identifier; and providing the digital content item on the computing device.

Abstract: The present disclosure relates to generating a complex entity index based on a combination of atomic and deep learned attributes associated with instances of a complex entity. For example, systems described herein generate a multi-dimensional representation of entity instances based on evaluation of digital content associated with the respective entity instances. Systems described herein further generate an index representation in which similarity of entity instances are illustrated and presented via an interactive presentation that enables a user to traverse instances of an entity to observe similarities and differences between instances of an entity that have similar embeddings to one another within a multi-dimensional index space.

Abstract: A data processing system implements obtaining query parameters for a query for content items in a datastore, the query parameters including attributes of content items for which a search is to be conducted; obtaining a first set of content items from a content datastore based on the query parameters; analyzing the first set of content items using a first machine learning model trained to generate relevant content information that identifies a plurality of relevant content items included in the first set of content items; and analyzing the plurality of relevant content items using a second machine learning model configured to output novel content information, the novel content information including a plurality of content items predicted to be relevant and novel, the novel content information ranking the plurality of content items predicted to be relevant and novel based on a novelty score associated with each respective content item.

Abstract: An example system includes a stand configured to position a computing device proximate to a physical activity surface. The system further includes a video capture device, a detector, and an activity application. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically interactable with by a user. The detector is executable to detect motion in the activity scene based on the processing and, responsive to detecting the motion, process the video stream to detect one or more interface objects included in the activity scene of the physical activity surface. The activity application is executable to present virtual information on a display of the computing device based on the one or more detected interface objects.

Abstract: A data processing system implements obtaining query parameters for a query for content items in a datastore, the query parameters including attributes of content items for which a search is to be conducted; obtaining a first set of content items from a content datastore based on the query parameters; analyzing the first set of content items using a first machine learning model trained to generate relevant content information that identifies a plurality of relevant content items included in the first set of content items; and analyzing the plurality of relevant content items using a second machine learning model configured to output novel content information, the novel content information including a plurality of content items predicted to be relevant and novel, the novel content information ranking the plurality of content items predicted to be relevant and novel based on a novelty score associated with each respective content item.

Abstract: An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.

Abstract: A computer implemented method includes receiving an image that includes a type of object, segmenting the object into multiple segments via a trained segmentation machine learning model, and inputting the segments into multiple different attribute extraction models to extract different types of attributes from each of the multiple segments.

Abstract: The present disclosure relates to selectively analyzing digital content items from a social networking system to generate a searchable graph object that facilitates visualization of correlations between entities of interest (e.g., brands, products, services, companies), concepts (e.g., domain-specific and/or general terms), hashtags, and other output classes. For example, systems described herein involve collecting images and/or videos that are publicly available via a social networking platform and evaluating content and metadata thereof to identify entities of interest therein and predict various concepts therein to generate a graph object that provides a searchable matrix. The systems herein provide a mechanism for processing a graph query that presents a relational graph showing correlations between the query and the various output classes.

Abstract: Systems and methods are provided for building and training machine learning models configured to generate in-domain embeddings and perform multimodal analysis inside the same domain. The models include a first encoder trained to receive input from one or more entities represented in a first modality and to encode the one or more entities in the first modality, such that the first encoder is configured to output a first set of embeddings. The models also include a second encoder trained to receive input from one or more entities represented in the second modality and to encode the one or more entities in the second modality, such that the second encoder is configured to output a second set of embeddings. The models also include a projection layer configured to project the first set of embeddings and the second set of embeddings to a shared contrastive space.

Abstract: The present disclosure relates to generating a complex entity index based on a combination of atomic and deep learned attributes associated with instances of a complex entity. For example, systems described herein generate a multi-dimensional representation of entity instances based on evaluation of digital content associated with the respective entity instances. Systems described herein further generate an index representation in which similarity of entity instances are illustrated and presented via an interactive presentation that enables a user to traverse instances of an entity to observe similarities and differences between instances of an entity that have similar embeddings to one another within a multi-dimensional index space.

Abstract: An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.

Abstract: A computer implemented method includes rendering a document page as an image; detecting tables, columns, and other associated table objects within the image via one or more table recognition models that model objects in the image as overlapping bounding boxes; transforming the set of objects into a structured representation of the table; extracting data from the objects into the structured representation; and exporting the table into the desired output format.

Abstract: A method and system for organizing a virtual classroom session. In an example implementation, a method includes, receiving first media data including a first user media stream depicting a first user and a first workspace media stream depicting a first physical activity scene that is proximate a first computing device of the first user and a second media data including a second user media stream depicting a second user and a second workspace media stream depicting a second physical activity scene that is proximate a second computing device of the second user, generating a graphical virtual meeting user interface and providing the graphical virtual meeting user interface for display.

Abstract: An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.