Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a machine learning model that has been trained through reinforcement learning to select a content item. One of the methods includes receiving first data characterizing a first context in which a first content item may be presented to a first user in a presentation environment; and providing the first data as input to a long-term engagement machine learning model, the model having been trained through reinforcement learning to: receive a plurality of inputs, and process each of the plurality of inputs to generate a respective engagement score for each input that represents a predicted, time-adjusted total number of selections by the respective user of future content items presented to the respective user in the presentation environment if the respective content item is presented in the respective context.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a machine learning model that has been trained through reinforcement learning to select a content item. One of the methods includes receiving first data characterizing a first context in which a first content item may be presented to a first user in a presentation environment; and providing the first data as input to a long-term engagement machine learning model, the model having been trained through reinforcement learning to: receive a plurality of inputs, and process each of the plurality of inputs to generate a respective engagement score for each input that represents a predicted, time-adjusted total number of selections by the respective user of future content items presented to the respective user in the presentation environment if the respective content item is presented in the respective context.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a machine learning model that has been trained through reinforcement learning to select a content item. One of the methods includes receiving first data characterizing a first context in which a first content item may be presented to a first user in a presentation environment; and providing the first data as input to a long-term engagement machine learning model, the model having been trained through reinforcement learning to: receive a plurality of inputs, and process each of the plurality of inputs to generate a respective engagement score for each input that represents a predicted, time-adjusted total number of selections by the respective user of future content items presented to the respective user in the presentation environment if the respective content item is presented in the respective context.

Abstract: Inferring query intent in information retrieval is described. In an example reformulations of an initial query by a user are used to create a query neighborhood. In the example, the query neighborhood is used to identify a set of possibly related queries. First and higher order reformulations of the initial query may be used to expand the query neighborhood. In an example precision can be improved by reducing the query neighborhood to more closely related queries for example, two queries can be connected if they are often clicked for the same document. In an example two queries can be connected using a random walk and all pairs of queries that are not connected by a random walk of less than a fixed threshold are removed. The connected queries can be used to form clusters and weights can be applied in order to determine the most likely related queries.

Abstract: Content-based information retrieval is described. In an example, a query item such as an image, document, email or other item is presented and items with similar content are retrieved from a database of items. In an example, each time a query is presented, a classifier is formed based on that query and using a training set of items. For example, the classifier is formed in real-time and is formed in such a way that a limit on the proportion of the items in the database that will be retrieved is set. In an embodiment, the query item is analyzed to identify tokens in that item and subsets of those tokens are selected to form the classifier. For example, the subsets of tokens are combined using Boolean operators in a manner which is efficient for searching on particular types of database.

Abstract: Inferring query intent in information retrieval is described. In an example reformulations of an initial query by a user are used to create a query neighborhood. In the example, the query neighborhood is used to identify a set of possibly related queries. First and higher order reformulations of the initial query may be used to expand the query neighborhood. In an example precision can be improved by reducing the query neighborhood to more closely related queries for example, two queries can be connected if they are often clicked for the same document. In an example two queries can be connected using a random walk and all pairs of queries that are not connected by a random walk of less than a fixed threshold are removed. The connected queries can be used to form clusters and weights can be applied in order to determine the most likely related queries.

Abstract: An information retrieval system is described for retrieving a list of documents such as web pages or other items from a document index in response to a user query. In an embodiment a prediction engine is used to predict both explicit relevance information such as judgment labels and implicit relevance information such as click data. In an embodiment the predicted relevance information is applied to a stored utility function that describes user satisfaction with a search session. This produces utility scores for proposed lists of documents. Using the utility scores one of the lists of documents is selected. In this way different sources of relevance information are combined into a single information retrieval system in a principled and effective manner which gives improved performance.

Abstract: Information retrieval using query-document pair information is described. In an embodiment, a click record is accessed having information about queries and documents where user clicks have been observed for query-document pairs. A click graph is either formed or accessed. This has nodes connected by edges, each node representing any of a document and a query and each edge representing at least one observed click. Given at least one first node in the click graph, a similarity measure is determined between that first node and each of one or more second nodes. The second nodes are then ranked on the basis of the similarity measure results and the ranking is used to retrieve information from the click record.

Abstract: Content-based information retrieval is described. In an example, a query item such as an image, document, email or other item is presented and items with similar content are retrieved from a database of items. In an example, each time a query is presented, a classifier is formed based on that query and using a training set of items. For example, the classifier is formed in real-time and is formed in such a way that a limit on the proportion of the items in the database that will be retrieved is set. In an embodiment, the query item is analyzed to identify tokens in that item and subsets of those tokens are selected to form the classifier. For example, the subsets of tokens are combined using Boolean operators in a manner which is efficient for searching on particular types of database.

Abstract: We set out a graphical model for describing probability distributions over labeled partitions of an undirected graph which are conditioned on observed data. We show how to efficiently perform exact inference in these models, by exploiting the structure of the graph and adapting the sum-product and max-product algorithms. The method can be used for partitioning and labeling hand-drawn ink fragments, image data, speech data and natural language data amongst other types of data elements. A significant performance increase is obtained by labeling and partitioning simultaneously. It is also possible to partition without labeling.

Abstract: An information retrieval system is described for retrieving a list of documents such as web pages or other items from a document index in response to a user query. In an embodiment a prediction engine is used to predict both explicit relevance information such as judgment labels and implicit relevance information such as click data. In an embodiment the predicted relevance information is applied to a stored utility function that describes user satisfaction with a search session. This produces utility scores for proposed lists of documents. Using the utility scores one of the lists of documents is selected. In this way different sources of relevance information are combined into a single information retrieval system in a principled and effective manner which gives improved performance.

Abstract: Digital ink strokes may be fragmented to form a training data set. A neighborhood graph may be formed as a plurality of connected nodes. Relevant features of the training data may be determined in each fragment such as local site features, interaction features, and/or part-label interaction features. Using a conditional random field which may include a hidden random field modeling parameters may be developed to provide a training model to determine a posterior probability of the labels given observed data. In this manner, the training model may be used to predict a label for an observed ink stroke. The modeling parameters may be learned from only a portion of the set of ink strokes in an unsupervised way. For example, many compound objects may include compositional parts. In some cases, appropriate compositional parts may be discovered or inferred during training of the model based on the training data.

Abstract: Information retrieval using query-document pair information is described. In an embodiment, a click record is accessed having information about queries and documents where user clicks have been observed for query-document pairs. A click graph is either formed or accessed. This has nodes connected by edges, each node representing any of a document and a query and each edge representing at least one observed click. Given at least one first node in the click graph, a similarity measure is determined between that first node and each of one or more second nodes. The second nodes are then ranked on the basis of the similarity measure results and the ranking is used to retrieve information from the click record.

Abstract: We set out a graphical model for describing probability distributions over labeled partitions of an undirected graph which are conditioned on observed data. We show how to efficiently perform exact inference in these models, by exploiting the structure of the graph and adapting the sum-product and max-product algorithms. The method can be used for partitioning and labeling hand-drawn ink fragments, image data, speech data and natural language data amongst other types of data elements. A significant performance increase is obtained by labeling and partitioning simultaneously. It is also possible to partition without labeling.

Abstract: A Bayesian approach to training in conditional random fields takes a prior distribution over the modeling parameters of interest. These prior distributions may be used to generate an approximate form of a posterior distribution over the parameters, which may be trained with example or training data. Automatic relevance determination (ARD) may be integrated in the training to automatically select relevant features of the training data. From the trained posterior distribution of the parameters, a posterior distribution over the parameters based on the training data and the prior distributions over parameters may be approximated to form a training model. Using the developed training model, a given image may be evaluated by integrating over the posterior distribution over parameters to obtain a marginal probability distribution over the labels given that observational data.

Abstract: Digital ink strokes may be fragmented to form a training data set. A neighborhood graph may be formed as a plurality of connected nodes. Relevant features of the training data may be determined in each fragment such as local site features, interaction features, and/or part-label interaction features. Using a conditional random field which may include a hidden random field modeling parameters may be developed to provide a training model to determine a posterior probability of the labels given observed data. In this manner, the training model may be used to predict a label for an observed ink stroke. The modeling parameters may be learned from only a portion of the set of ink strokes in an unsupervised way. For example, many compound objects may include compositional parts. In some cases, appropriate compositional parts may be discovered or inferred during training of the model based on the training data.