Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having a first classification layer, a second classification layer, and an encoder that feeds into the first classification layer and the second classification layer. The example method also involves obtaining first training examples having explicit labels and second training examples having inferred labels. The inferred labels are based at least on actions associated with the second training examples. The example method also involves training the machine learning model using the first training examples and the second training examples using a training objective that considers first training loss of the first classification layer for the explicit labels and second training loss of the second classification layer for the inferred labels. The method also involves outputting a trained machine learning model having the encoder and the first classification layer.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having a first classification layer, a second classification layer, and an encoder that feeds into the first classification layer and the second classification layer. The example method also involves obtaining first training examples having explicit labels and second training examples having inferred labels. The inferred labels are based at least on actions associated with the second training examples. The example method also involves training the machine learning model using the first training examples and the second training examples using a training objective that considers first training loss of the first classification layer for the explicit labels and second training loss of the second classification layer for the inferred labels. The method also involves outputting a trained machine learning model having the encoder and the first classification layer.

Abstract: A personalized natural language processing system tokenizes a plurality of sets of raw text data to generate a plurality of sets of tokenized text data for the plurality of users, respectively. The tokenized text data includes a sequence of tokens corresponding to the raw text data, the tokens at least identifying distinct words or portions of words in the raw text. The system appends predetermined user-specific tokens to the sets of tokenized text data from the users, respectively. Each predetermined user-specific token corresponds to one of the users. The system processes the sets of tokenized text data using the NLP model in accordance with the appended predetermined user-specific tokens to predict a personalized classification for the sets of tokenized text data from each of the users, and outputs the personalized classifications of the tokenized text data for each of the users.

Abstract: Systems and methods are provided that automatically process message input and provide action responses according to the processing results. The automatic action response system may leverage at least one machine-learning algorithm that is trained using a dataset. The provided action responses may comprise of default action responses and/or intelligent action responses that are based at least in part on prior conversational data between a user and a sender. Some intelligent action responses may include text-based replies, which eliminate the need for a user to type a reply on a device screen, thereby saving previous time, conserving device battery life, and preserving the integrity of the device hardware. A portion of a message may be highlighted manually by a user or automatically by the action response system to initiate the automatic action response system. In this way, a more efficient and productive user experience across various devices and applications is achieved.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having a first classification layer, a second classification layer, and an encoder that feeds into the first classification layer and the second classification layer. The example method also involves obtaining first training examples having explicit labels and second training examples having inferred labels. The inferred labels are based at least on actions associated with the second training examples. The example method also involves training the machine learning model using the first training examples and the second training examples using a training objective that considers first training loss of the first classification layer for the explicit labels and second training loss of the second classification layer for the inferred labels. The method also involves outputting a trained machine learning model having the encoder and the first classification layer.

Abstract: Systems and methods are provided that automatically process message input and provide action responses according to the processing results. The automatic action response system may leverage at least one machine-learning algorithm that is trained using a dataset. The provided action responses may comprise of default action responses and/or intelligent action responses that are based at least in part on prior conversational data between a user and a sender. Some intelligent action responses may include text-based replies, which eliminate the need for a user to type a reply on a device screen, thereby saving previous time, conserving device battery life, and preserving the integrity of the device hardware. A portion of a message may be highlighted manually by a user or automatically by the action response system to initiate the automatic action response system. In this way, a more efficient and productive user experience across various devices and applications is achieved.

Abstract: Systems and methods are provided that automatically process message input and provide action responses according to the processing results. The automatic action response system may leverage at least one machine-learning algorithm that is trained using a dataset. The provided action responses may comprise of default action responses and/or intelligent action responses that are based at least in part on prior conversational data between a user and a sender. Some intelligent action responses may include text-based replies, which eliminate the need for a user to type a reply on a device screen, thereby saving previous time, conserving device battery life, and preserving the integrity of the device hardware. A portion of a message may be highlighted manually by a user or automatically by the action response system to initiate the automatic action response system. In this way, a more efficient and productive user experience across various devices and applications is achieved.

Abstract: Crowdsourcing systems with machine learning are described. Specifically, item-label inference methods and systems are presented, for example, to provide aggregated answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples, an item-label inference system infers variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, an item-label inference system provides aggregated labels while considering the inferred worker characteristics and the inferred characteristics of the worker communities. In examples the item-label inference system provides uncertainty information associated with the inference results for selecting workers and generating future tasks.

Abstract: Systems and methods are provided for determining whether a user has deferred one or more emails. More specifically, a system and method may determine whether an email is likely to have been deferred by a user, perform at least one action on the email determined likely to have been deferred, determine a mode for providing an indication to the user to follow-up with the email determined likely to have been deferred, and cause an indication specific to the email determined likely to have been deferred to be provided to the user. In some instances, the notifications are based on a device associated with the user and/or may be included in at least one of a task management application and/or a calendar application.

Abstract: Various technologies related to generating and applying content retrieval rules are described herein. A content retrieval rule maps a combination of a query and a context to one of a query reformulation or content. The content retrieval rule is learned from search logs of a search engine, and is applied when the query having the context is received at the search engine.

Abstract: Various technologies related to generating and applying content retrieval rules are described herein. A content retrieval rule maps a combination of a query and a context to one of a query reformulation or content. The content retrieval rule is learned from search logs of a search engine, and is applied when the query having the context is received at the search engine.

Abstract: Methods and systems are provided for providing alternative query suggestions. For example, a spoken natural language expression may be received and converted to a textual query by a speech recognition component. The spoken natural language expression may include one or more words, terms, and/or phrases. A phonetically confusable segment of the textual query may be identified by a classifier component. The classifier component may determine at least one alternative query based on identifying at least the phonetically confusable segment of the textual query. The classifier may further determine whether to suggest the at least one alternative query based on whether the at least one alternative query is sensical and/or useful. When it is determined to suggest the at least one alternative query, the at least one alternative query may be provided to and displayed on a user interface display.

Abstract: Systems and methods are provided that automatically process message input and provide action responses according to the processing results. The automatic action response system may leverage at least one machine-learning algorithm that is trained using a dataset. The provided action responses may comprise of default action responses and/or intelligent action responses that are based at least in part on prior conversational data between a user and a sender. Some intelligent action responses may include text-based replies, which eliminate the need for a user to type a reply on a device screen, thereby saving previous time, conserving device battery life, and preserving the integrity of the device hardware. A portion of a message may be highlighted manually by a user or automatically by the action response system to initiate the automatic action response system. In this way, a more efficient and productive user experience across various devices and applications is achieved.

Abstract: A set of content items, such as web pages, are identified in response to a query generated by a user. The Identified content items are initially ranked using a ranking scheme. User-interaction data that describes preferences that the user may have towards some of the ranked content items is received. In order to personalize the ranking of the content items for the user, the user-interaction data is used to re-rank the ranked content items in a way that favors content items that are preferred by the user, while also preserving the initial broadly applicable ranking with respect to content items that are not preferred or that are equally preferred by the user.

Abstract: Crowdsourcing systems with machine learning are described. Specifically, item-label inference methods and systems are presented, for example, to provide aggregated answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples, an item-label inference system infers variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, an item-label inference system provides aggregated labels while considering the inferred worker characteristics and the inferred characteristics of the worker communities. In examples the item-label inference system provides uncertainty information associated with the inference results for selecting workers and generating future tasks.

Abstract: Crowdsourcing systems with machine learning are described, for example, to aggregate answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples a machine learning system jointly learns variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, the machine learning system learns aggregated labels. In examples learnt variables describing characteristics of an individual crowd worker are related, by addition of noise, to learnt variables describing characteristics of a community of which the individual is a member. In examples the crowdsourcing system uses the learnt variables describing characteristics of individual workers and of communities of workers for any one or more of: active learning, targeted training of workers, targeted issuance of tasks, calculating and issuing rewards.

Abstract: A set of content items, such as web pages, are identified in response to a query generated by a user. The Identified content items are initially ranked using a ranking scheme. User-interaction data that describes preferences that the user may have towards some of the ranked content items is received. In order to personalize the ranking of the content items for the user, the user-interaction data is used to re-rank the ranked content items in a way that favors content items that are preferred by the user, while also preserving the initial broadly applicable ranking with respect to content items that are not preferred or that are equally preferred by the user.

Abstract: Merging search results is required, for example, where an information retrieval system issues a query to multiple sources and obtains multiple results lists. In an embodiment a search engine at an Enterprise domain sends a query to the Enterprise search engine and also to a public Internet search engine. In embodiments, results lists obtained from different sources are merged using a merging model which is learnt using a machine learning process and updates when click-through data is observed for example. In examples, user information available in the Enterprise domain is used to influence the merging process to improve the relevance of results. In some examples, the user information is used for query modification. In an embodiment a user is able to impersonate a user of a specified group in order to promote particular results.

Abstract: Methods and systems are provided for providing alternative query suggestions. For example, a spoken natural language expression may be received and converted to a textual query by a speech recognition component. The spoken natural language expression may include one or more words, terms, and/or phrases. A phonetically confusable segment of the textual query may be identified by a classifier component. The classifier component may determine at least one alternative query based on identifying at least the phonetically confusable segment of the textual query. The classifier may further determine whether to suggest the at least one alternative query based on whether the at least one alternative query is sensical and/or useful. When it is determined to suggest the at least one alternative query, the at least one alternative query may be provided to and displayed on a user interface display.

Abstract: A system and method are provided for detecting entity information contained within search results. The detected entity information can be used to determine a category of entity as well as a specific entity within the search results. Entity information can be extracted from the documents associated with the search results. This information can be used as part of the information for an entity card, which can be displayed to a user in conjunction with and/or in place of the search results.