Abstract: Systems, methods and apparatus for learning parsing rules and argument identification from crowdsourcing of proposed command inputs. Crowdsourcing techniques to generate rules for parsing input sentences. A parse is used to determine whether the input sentence invokes a specific action, and if so, what arguments are to be passed to the invocation of the action.

Abstract: Methods, systems, and apparatus for generating data describing context clusters and context cluster probabilities, wherein each context cluster includes query inputs based on the input context for each of the query inputs and the content described by each query input, and each context cluster probability indicates a probability that at a query input that belongs to the context cluster will be selected by the user, receiving, from a user device, an indication of a user event that includes data indicating a context of the user device, selecting as a selected context cluster, based on the context cluster probabilities for each of the context clusters and the context of the user device, a context cluster for selection input by the user device, and providing, to the user device, data that causes the user device to display a context cluster selection input that indicates the selected context cluster for user selection.

Abstract: A method includes accessing command sentences stored in a data store, wherein each command sentence is a collection of n-grams and each command sentence includes at least one n-gram that is a non-terminal n-gram that maps to a non-terminal type, and wherein the command sentences include non-terminal n-grams that collectively map to a plurality of different non-terminal types; for each of the non-terminal types: identifying n-gram spans; determining clusters of the n-gram spans, each cluster including n-gram spans meeting a measure of similarity of n-grams spans that belong to the cluster; and for each cluster of n-gram spans, determining, from the n-gram spans belonging to the cluster, a new non-terminal type to which the terminal n-grams of the n-gram spans map.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for rating participants based on emotive feedback. In one aspect, a method includes identifying a first resource that includes a form associated with a particular action, identifying, using one or more computers, a second resource that references the form of the first resource, identifying anchor text in the second resource that references the form, and adding the identified anchor text to a set of phrases that is used for generating an action initiation grammar associated with the particular action.

Abstract: A method identifies pairs of first and second command inputs from respective user device sessions for which the first and second operation data are indicative of a first operation failure and a second operation success. The first operation data indicate a first operation performed on data from a first resource property in response to the first command input, and the second operation data indicate a second operation performed on data from a second resource property in response to the second command input. They system determines, from the identified pairs of first and second command inputs, command inputs for which a parsing rule that is associated with the second operation is to be generated.

Abstract: Systems, methods and apparatus for learning parsing rules and argument identification from crowdsourcing of proposed command inputs are disclosed. Crowdsourcing techniques are used to generate rules for parsing input sentences. A parse is used to determine whether the input sentence invokes a specific action, and if so, what arguments are to be passed to the invocation of the action.

Abstract: A method identifies pairs of first and second command inputs from respective user device sessions for which the first and second operation data are indicative of a first operation failure and a second operation success. The first operation data indicate a first operation performed on data from a first resource property in response to the first command input, and the second operation data indicate a second operation performed on data from a second resource property in response to the second command input. They system determines, from the identified pairs of first and second command inputs, command inputs for which a parsing rule that is associated with the second operation is to be generated.

Abstract: A language processing system identifies sequential command inputs in user session data stored in logs. Each sequence command input is a first command input followed by a second command input. The system determines user actions in response to each command input. For the second command input, an action was taken at the user device in response to the command input, and there is no parsing rule associated with the action that parses to the first command input. If there is a sufficient co-occurrence of the first and second command inputs and the resulting action in the logs, then a parsing rule for the action may be augmented with a rule for the first command input.

Abstract: The present disclosure relates to evaluating different semantic interpretations of a search query. One example method includes obtaining a set of search results for a particular search query submitted to a search engine; obtaining a set of semantic interpretations for the particular search query; obtaining, for each semantic interpretation of the set, a canonical search query; generating a modified search query based at least in part on the particular search query and the canonical search query for the semantic interpretation; obtaining a set of search results for the modified search query for the semantic interpretation; and determining, for each semantic interpretation of the set, a degree of similarity between (i) the set of search results of the modified search query for the semantic interpretation, and (ii) the set of search results for the particular search query.

Abstract: A language processing system uses a lattice parser that semantically parses a command input represented by a lattice. The parser receives a hypotheses space of outputs as encoded in a lattice. Annotations of the input are projected back into the lattice and then lattice parsing is performed to rectify with the annotations. Parsing rules are applied to path fragments in the lattice. The rules that successfully parse from the start node to the end node of the lattice are used to determine whether the command input sentence invokes a specific action, and if so, what arguments are to be passed to the invocation of the action.

Abstract: A language processing system identifies, from log data, command inputs that parsed to a parsing rule associated with an action. If the command input has a signal indicative of user satisfaction, where the signal is derived from data that is not generated from performance of the action (e.g., user interactions with data provided in response to the performance of another, different action; resources identified in response to the performance of another, different action having a high quality score; etc.), then exception data is generated for the parsing rule. The exception data specifies the particular instance of the sentence parsed by the parsing rule, and precludes invocation of the action associated with the rule.

Abstract: A system determines search hypotheses for a search query, each search hypothesis defining a search type and respectively corresponding to a resource corpus of a type that matches the search type; for each search hypothesis, generate a hypothesis search query based on the search query and the search type and submits the hypothesis search query to a search service to determine a search hypothesis score, and for each search hypothesis having a search hypothesis score meeting a search hypothesis threshold, providing search results for the search operation performed for the hypothesis search query determined for the search hypothesis; and for each search hypothesis not having a search hypothesis score meeting a search hypothesis threshold, not providing search results for the search operation performed for the hypothesis search query determined for the search hypothesis.

Abstract: A language processing system uses annotation services that are external to the language processing system to identify n-grams that identify entities in an input sentence. The n-grams are annotated by the annotation services. The annotations are used to determine which n-grams, if any, correspond to instances of an entity type (e.g., values for a variable or terminals for a non-terminal). After determining which n-grams correspond to entity types, parse initializations are generated for parsing rules and parses for each rule are attempted. The rules that successfully parse are used to determine whether the input sentence invokes a specific action, and if so, what arguments are to be passed to the invocation of the action.

Abstract: A language processing system identifies first command input sentences that do not successfully parse by any parsing rule in a set of parsing rules. Each of the parsing rules is associated with an action, and a user device performs the action associated with a parsing rule in response to an input sentence being successfully parsed by the parsing rule. For each of these identified first sentences, the system determines whether the first input sentence has an underserving signal that is indicative of one or more actions being underserved. If the first sentence has the underserving signal, then the first sentence is selected as a candidate input sentence. Each candidate input sentence is provided to an action analysis processes that determines whether a candidate input sentence is to be associated with one action, and upon a positive determination generates a parsing rule for the candidate input sentence.

Abstract: Systems, methods and apparatus for learning parsing rules and argument identification from crowdsourcing of proposed command inputs are disclosed. Crowdsourcing techniques are used to generate rules for parsing input sentences. A parse is used to determine whether the input sentence invokes a specific action, and if so, what arguments are to be passed to the invocation of the action.

Abstract: A language processing system identifies, from log data, command inputs that parsed to a parsing rule associated with an action. If the command input has a signal indicative of user satisfaction, where the signal is derived from data that is not generated from performance of the action (e.g., user interactions with data provided in response to the performance of another, different action; resources identified in response to the performance of another, different action having a high quality score; etc.), then exception data is generated for the parsing rule. The exception data specifies the particular instance of the sentence parsed by the parsing rule, and precludes invocation of the action associated with the rule.

Abstract: A method includes accessing command sentences stored in a data store, wherein each command sentence is a collection of n-grams and each command sentence includes at least one n-gram that is a non-terminal n-gram that maps to a non-terminal type, and wherein the command sentences include non-terminal n-grams that collectively map to a plurality of different non-terminal types; for each of the non-terminal types: identifying n-gram spans; determining clusters of the n-gram spans, each cluster including n-gram spans meeting a measure of similarity of n-grams spans that belong to the cluster; and for each cluster of n-gram spans, determining, from the n-gram spans belonging to the cluster, a new non-terminal type to which the terminal n-grams of the n-gram spans map.