Abstract: A system and method for automatically adjusting computing resources provisioned for a computer service or application by applying historical resource usage data to a predictive model to generate predictive resource usage. The predictive resource usage is then simulated for various service configurations, determining scaling requirements and resource wastage for each configuration. A cost value is generated based on the scaling requirement and resource wastage, with the cost value for each service configuration used to automatically select a configuration to apply to the service. Alternatively, the method for automatically adjusting computer resources provisioned for a service may include receiving resource usage data of the service, applying it to a linear quadratic regulator (LQR) to find an optimal stationary policy (treating the resource usage data as states and resource-provisioning variables as actions), and providing instructions for configuring the service based on the optimal stationary policy.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for generating semantic scene graphs for digital images using an external knowledgebase for feature refinement. For example, the disclosed system can determine object proposals and subgraph proposals for a digital image to indicate candidate relationships between objects in the digital image. The disclosed system can then extract relationships from an external knowledgebase for refining features of the object proposals and the subgraph proposals. Additionally, the disclosed system can generate a semantic scene graph for the digital image based on the refined features of the object/subgraph proposals. Furthermore, the disclosed system can update/train a semantic scene graph generation network based on the generated semantic scene graph. The disclosed system can also reconstruct the image using object labels based on the refined features to further update/train the semantic scene graph generation network.

Abstract: A multimodal recommendation identification system analyzes data describing a sequence of past content item interactions to generate a recommendation for a content item for a user. An indication of the recommended content item is provided to a website hosting system or recommendation system so that the recommended content item is displayed or otherwise presented to the user. The multimodal recommendation identification system identifies a content item to recommend to the user by generating an encoding that encodes identifiers of the sequence of content items the user has interacted with and generating encodings that encode multimodal information for content items in the sequence of content items the user has interacted with. An aggregated information encoding for a user based on these encodings and a system analyzes the content item sequence encoding and interaction between the content item sequence encoding and the multiple modality encodings to generate the aggregated information encoding.

Abstract: Adapting a machine learning model to process data that differs from training data used to configure the model for a specified objective is described. A domain adaptation system trains the model to process new domain data that differs from a training data domain by using the model to generate a feature representation for the new domain data, which describes different content types included in the new domain data. The domain adaptation system then generates a probability distribution for each discrete region of the new domain data, which describes a likelihood of the region including different content described by the feature representation. The probability distribution is compared to ground truth information for the new domain data to determine a loss function, which is used to refine model parameters. After determining that model outputs achieve a threshold similarity to the ground truth information, the model is output as a domain-agnostic model.

Abstract: In some embodiments, a data visualization system detects insights from a dataset and computes insight scores for respective insights. The data visualization system further computes insight type scores, from the insight scores, for insight types in the detected insights. The data visualization system determines a selected insight type for the dataset having a higher insight type score than unselected insight types and determines, for the selected insight type, a set of selected insights that have higher insight scores than unselected insights. The data visualization system determines insight visualizations for the set of selected insights and generates, for inclusion in a user interface of the data visualization system, selectable interface elements configured for invoking an editing tool for updating the determined insight visualizations from the dataset. The selectable interface elements are arranged in the user interface according to the insight scores of the set of selected insights.

Abstract: Systems and methods for item recommendation are described. Embodiments identify a sequence of items selected by a user, embed each item of the sequence of items to produce item embeddings having a reduced number of dimensions, predict a next item based on the item embeddings using a recommendation network, wherein the recommendation network includes a sequential encoder trained based at least in part on a sampled softmax classifier, and wherein predicting the next item represents a prediction that the user will interact with the next item, and provide a recommendation to the user, wherein the recommendation includes the next item.

Abstract: Methods and systems are provided for generating interpretable user modeling system. The interpretable user modeling system can use an intent neural network to implement one or more tasks. The intent neural network can bridge a semantic gap between log data and human language by leveraging tutorial data to understand user logs in a semantically meaningful way. A memory unit of the intent neural network can capture information from the tutorial data. Such a memory unit can be queried to identify human readable sentences related to actions received by the intent neural network. The human readable sentences can be used to interpret the user log data in a semantically meaningful way.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for generating semantic scene graphs for digital images using an external knowledgebase for feature refinement. For example, the disclosed system can determine object proposals and subgraph proposals for a digital image to indicate candidate relationships between objects in the digital image. The disclosed system can then extract relationships from an external knowledgebase for refining features of the object proposals and the subgraph proposals. Additionally, the disclosed system can generate a semantic scene graph for the digital image based on the refined features of the object/subgraph proposals. Furthermore, the disclosed system can update/train a semantic scene graph generation network based on the generated semantic scene graph. The disclosed system can also reconstruct the image using object labels based on the refined features to further update/train the semantic scene graph generation network.

Abstract: Methods and systems disclosed herein relate generally to systems and methods for generating visual relationship graphs that identify relationships between objects depicted in an image. A vision-language application uses transformer encoders to generate a graph structure, in which the graph structure represents a dependency between a first region and a second region of an image. The dependency indicates that a contextual representation of the first region was derived, at least in part, by processing the second region. The contextual representation identifies a predicted identity of an image object depicted in the first region. The predicted identity is determined at least in part by identifying a relationship between the first region and other data objects associated with various modalities.

Abstract: Systems and methods for lifelong schema matching are described. The systems and methods include receiving data comprising a plurality of information categories, classifying each information category according to a schema comprising a plurality of classes, wherein the classification is performed by a neural network classifier trained based on a lifelong learning technique using a plurality of exemplar training sets, wherein each of the exemplar training sets includes a plurality of examples corresponding to one of the classes, and wherein the examples are selected based on a metric indicating how well each of the examples represents the corresponding class, and adding the data to a database based on the classification, wherein the database is organized according to the schema.

Abstract: The technology described herein is directed to a reinforcement learning based framework for training a natural media agent to learn a rendering policy without human supervision or labeled datasets. The reinforcement learning based framework feeds the natural media agent a training dataset to implicitly learn the rendering policy by exploring a canvas and minimizing a loss function. Once trained, the natural media agent can be applied to any reference image to generate a series (or sequence) of continuous-valued primitive graphic actions, e.g., sequence of painting strokes, that when rendered by a synthetic rendering environment on a canvas, reproduce an identical or transformed version of the reference image subject to limitations of an action space and the learned rendering policy.

Abstract: A system and method for automatically adjusting computing resources provisioned for a computer service or application by applying historical resource usage data to a predictive model to generate predictive resource usage. The predictive resource usage is then simulated for various service configurations, determining scaling requirements and resource wastage for each configuration. A cost value is generated based on the scaling requirement and resource wastage, with the cost value for each service configuration used to automatically select a configuration to apply to the service. Alternatively, the method for automatically adjusting computer resources provisioned for a service may include receiving resource usage data of the service, applying it to a linear quadratic regulator (LQR) to find an optimal stationary policy (treating the resource usage data as states and resource-provisioning variables as actions), and providing instructions for configuring the service based on the optimal stationary policy.

Abstract: Adapting a machine learning model to process data that differs from training data used to configure the model for a specified objective is described. A domain adaptation system trains the model to process new domain data that differs from a training data domain by using the model to generate a feature representation for the new domain data, which describes different content types included in the new domain data. The domain adaptation system then generates a probability distribution for each discrete region of the new domain data, which describes a likelihood of the region including different content described by the feature representation. The probability distribution is compared to ground truth information for the new domain data to determine a loss function, which is used to refine model parameters. After determining that model outputs achieve a threshold similarity to the ground truth information, the model is output as a domain-agnostic model.

Abstract: The present disclosure relates to a retrieval method including: generating a graph representing a set of users, items, and queries; generating clusters from the media items; generating embeddings for each cluster from embeddings of the items within the corresponding cluster; generating augmented query embeddings for each cluster from the embedding of the corresponding cluster and query embeddings of the queries; inputting the cluster embeddings and the augmented query embeddings to a layer of a graph convolutional network (GCN) to determine user embeddings of the users; inputting the embedding of the given user and a query embedding of the given query to a layer of the GCN to determine a user-specific query embedding; generating a score for each of the items based on the item embeddings and the user-specific query embedding; and presenting the items having the score exceeding a threshold.

Abstract: A domain alignment technique for cross-domain object detection tasks is introduced. During a preliminary pretraining phase, an object detection model is pretrained to detect objects in images associated with a source domain using a source dataset of images associated with the source domain. After completing the pretraining phase, a domain adaptation phase is performed using the source dataset and a target dataset to adapt the pretrained object detection model to detect objects in images associated with the target domain. The domain adaptation phase may involve the use of various domain alignment modules that, for example, perform multi-scale pixel/path alignment based on input feature maps or perform instance-level alignment based on input region proposals.

Abstract: Certain embodiments involve a method for generating a search result. The method includes processing devices performing operations including receiving a query having a text input by a joint embedding model trained to generate an image result. Training the joint embedding model includes accessing a set of images and textual information. Training further includes encoding the images into image feature vectors based on spatial features. Further, training includes encoding the textual information into textual feature vectors based on semantic information. Training further includes generating a set of image-text pairs based on matches between image feature vectors and textual feature vectors. Further, training includes generating a visual grounding dataset based on spatial information. Training further includes generating a set of visual-semantic joint embeddings by grounding the image-text pairs with the visual grounding dataset.

Abstract: Systems, methods, and non-transitory computer-readable media are disclosed for generating generate explanatory paths for column annotations determined using a knowledge graph and a deep representation learning model. For instance, the disclosed systems can utilize a knowledge graph to generate an explanatory path for a column label determination from a deep representation learning model. For example, the disclosed systems can identify a column and determine a label for the column using a knowledge graph (e.g., a representation of a knowledge graph) that includes encodings of columns, column features, relational edges, and candidate labels. Then, the disclosed systems can determine a set of candidate paths between the column and the determined label for the column within the knowledge graph. Moreover, the disclosed systems can generate an explanatory path by ranking and selecting paths from the set of candidate paths using a greedy ranking and/or diversified ranking approach.

Abstract: Systems, methods, and non-transitory computer-readable media are disclosed for dynamically determining schema labels for columns regardless of information availability within the columns. For example, the disclosed systems can identify a column that contains an arbitrary amount of information (e.g., a header-only column, a cell-only column, or a whole column). Additionally, the disclosed systems can generate a vector embedding for an arbitrary input column by selectively using a header neural network and/or a cell neural network based on whether the column includes a header label and/or whether the column includes a populated column cell. Furthermore, the disclosed systems can compare the column vector embedding to schema vector embeddings of candidate schema labels in a d-dimensional space to determine a schema label for the column.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for generating semantic scene graphs for digital images using an external knowledgebase for feature refinement. For example, the disclosed system can determine object proposals and subgraph proposals for a digital image to indicate candidate relationships between objects in the digital image. The disclosed system can then extract relationships from an external knowledgebase for refining features of the object proposals and the subgraph proposals. Additionally, the disclosed system can generate a semantic scene graph for the digital image based on the refined features of the object/subgraph proposals. Furthermore, the disclosed system can update/train a semantic scene graph generation network based on the generated semantic scene graph. The disclosed system can also reconstruct the image using object labels based on the refined features to further update/train the semantic scene graph generation network.

Abstract: Methods and systems are provided for generating interpretable user modeling system. The interpretable user modeling system can use an intent neural network to implement one or more tasks. The intent neural network can bridge a semantic gap between log data and human language by leveraging tutorial data to understand user logs in a semantically meaningful way. A memory unit of the intent neural network can capture information from the tutorial data. Such a memory unit can be queried to identify human readable sentences related to actions received by the intent neural network. The human readable sentences can be used to interpret the user log data in a semantically meaningful way.