Abstract: A hierarchical extraction model for a label hierarchy may be implemented by a weighted hierarchical state machine whose structure and/or weights are determined in part from a statistical distribution of label sequences as determined from training data. In accordance with various embodiments, the hierarchical state machine includes one or more non-cyclic directed chains of states representing at least a subset of the label sequences, and transitions weighted based at least in part on the statistical distribution.

Abstract: Domain-specific language understanding models that may be built, tested and improved quickly and efficiently are provided. Methods, systems and devices are provided that enable a developer to build user intent detection models, language entity extraction models, and language entity resolution models quickly and without specialized machine learning knowledge. These models may be built and implemented via single model systems that enable the models to be built in isolation or in an end-to-end pipeline system that enables the models to be built and improved in a simultaneous manner.

Abstract: Domain-specific language understanding models that may be built, tested and improved quickly and efficiently are provided. Methods, systems and devices are provided that enable a developer to build user intent detection models, language entity extraction models, and language entity resolution models quickly and without specialized machine learning knowledge. These models may be built and implemented via single model systems that enable the models to be built in isolation or in an end-to-end pipeline system that enables the models to be built and improved in a simultaneous manner.

Abstract: Domain-specific language understanding models that may be built, tested and improved quickly and efficiently are provided. Methods, systems and devices are provided that enable a developer to build user intent detection models, language entity extraction models, and language entity resolution models quickly and without specialized machine learning knowledge. These models may be built and implemented via single model systems that enable the models to be built in isolation or in an end-to-end pipeline system that enables the models to be built and improved in a simultaneous manner.

Abstract: A hierarchical extraction model for a label hierarchy may be implemented by a weighted hierarchical state machine whose structure and/or weights are determined in part from a statistical distribution of label sequences as determined from training data. In accordance with various embodiments, the hierarchical state machine includes one or more non-cyclic directed chains of states representing at least a subset of the label sequences, and transitions weighted based at least in part on the statistical distribution.

Abstract: Technologies are described herein for active machine learning. An active machine learning method can include initiating active machine learning through an active machine learning system configured to train an auxiliary machine learning model to produce at least one new labeled observation, refining a capacity of a target machine learning model based on the active machine learning, and retraining the auxiliary machine learning model with the at least one new labeled observation subsequent to refining the capacity of the target machine learning model. Additionally, the target machine learning model is a limited-capacity machine learning model according to the description provided herein.

Abstract: Methods, computer systems, computer-storage media, and graphical user interfaces are provided for visualizing a performance of a machine-learned model. An interactive graphical user interface includes an item representation display area that displays a plurality of item representations corresponding to a plurality of items processed by the machine-learned model. The plurality of item representations are arranged according to scores assigned to the plurality of items by the machine-learned model. Further, each of the plurality of item representations is visually configured to represent a label assigned to a corresponding item.

Abstract: Distribution displays for categories are provided which illuminate the distribution of continuous attributes over all cases in a category, and which provide a histogram of the population of the different states of categorical attributes. An array of such displays by attribute (in one dimension) and category (in another dimension) may be provided. Category diagram displays are also provided for visualizing the different categories, and their distributions, populations, and similarities. These are displayed through different shading of nodes and edges representing categories and the relationship between two categories, and through proximity of nodes.

Abstract: Domain-specific language understanding models that may be built, tested and improved quickly and efficiently are provided. Methods, systems and devices are provided that enable a developer to build user intent detection models, language entity extraction models, and language entity resolution models quickly and without specialized machine learning knowledge. These models may be built and implemented via single model systems that enable the models to be built in isolation or in an end-to-end pipeline system that enables the models to be built and improved in a simultaneous manner.

Abstract: Technologies are described herein for active machine learning. An active machine learning method can include initiating active machine learning through an active machine learning system configured to train an auxiliary machine learning model to produce at least one new labeled observation, refining a capacity of a target machine learning model based on the active machine learning, and retraining the auxiliary machine learning model with the at least one new labeled observation subsequent to refining the capacity of the target machine learning model. Additionally, the target machine learning model is a limited-capacity machine learning model according to the description provided herein.

Abstract: Methods, computer systems, computer-storage media, and graphical user interfaces are provided for visualizing a performance of a machine-learned model. An interactive graphical user interface includes an item representation display area that displays a plurality of item representations corresponding to a plurality of items processed by the machine-learned model. The plurality of item representations are arranged according to scores assigned to the plurality of items by the machine-learned model. Further, each of the plurality of item representations is visually configured to represent a label assigned to a corresponding item.

Abstract: One or more systems and/or techniques are provided for constructing a query classification index that can be used to classify a query into relevant categories. Where documents in an index are classified into one or more category predictions for a category hierarchy, classification metadata is generated for categories to which a document in the index has been classified. Further, the classification metadata is associated to the corresponding documents in the index. Additionally, a query of the index can be classified using the metadata associated to the documents in the index, and query results can be provided that are classified by the one or more categories identified by the classification of the query.

Abstract: Various technologies described herein pertain to smoothing budgets of advertisers in online advertising. Information that indicates respective budgets of advertisers for online advertising during a time period and bids for auctions during the time period from the advertisers can be received. Moreover, a determination concerning whether to either throttle an advertiser from an auction or permit the advertiser to participate in the auction can be effectuated for each of the auctions during the time period and for each of the advertisers. The determination satisfies equilibrium conditions between the advertisers based on the respective budgets of the advertisers and the bids of the auctions. Moreover, winning bids for the auctions during the time period are determined. The winning bids can be determined from the bids of the advertisers respectively permitted to participate in each of the auctions.

Abstract: Providing a market design for a peer-to-peer resource exchange system. Prices for a plurality of resources such as storage space, upload bandwidth, and download bandwidth are calculated and balanced based on previous resource prices, a supply of the resources, and a demand for the resources. Further, prices for operations such as storage and retrieval are determined such that a total of the payments to resource suppliers equals a total of the payments received from the resource consumers. In some embodiments, incoming data operation requests are allocated to the peers such that equilibrium among the peers is achieved.

Abstract: Counterfactual analysis can be performed “offline”, or “after the fact”, based on data collected during a trial in which random variations are applied to the output of the system whose parameters are to be the subject of the counterfactual analysis. A weighting factor can be derived and applied to data collected during the trial to emphasize that data obtained when the random variations most closely resembled the output that would be expected if counterfactual parameters were utilized to generate the output. If the counterfactual parameters being considered differ too much from the parameters under which the trial was conducted, the offline counterfactual analysis can estimate a direction and magnitude of the change of the system performance, as opposed to deriving a specific expected system performance value. In economic transactions, the random variations can be considered variations in the price paid by another party, thereby enabling derivation of their marginal cost.

Abstract: Counterfactual analysis can be performed “offline”, or “after the fact”, based on data collected during a trial in which random variations are applied to the output of the system whose parameters are to be the subject of the counterfactual analysis. A weighting factor can be derived and applied to data collected during the trial to emphasize that data obtained when the random variations most closely resembled the output that would be expected if counterfactual parameters were utilized to generate the output. If the counterfactual parameters being considered differ too much from the parameters under which the trial was conducted, the offline counterfactual analysis can estimate a direction and magnitude of the change of the system performance, as opposed to deriving a specific expected system performance value. In economic transactions, the random variations can be considered variations in the price paid by another party, thereby enabling derivation of their marginal cost.

Abstract: Providing a market design for a peer-to-peer resource exchange system. Prices for a plurality of resources such as storage space, upload bandwidth, and download bandwidth are calculated and balanced based on previous resource prices, a supply of the resources, and a demand for the resources. Further, prices for operations such as storage and retrieval are determined such that a total of the payments to resource suppliers equals a total of the payments received from the resource consumers. In some embodiments, incoming data operation requests are allocated to the peers such that equilibrium among the peers is achieved.

Abstract: Providing a market design for a peer-to-peer resource exchange system. Prices for a plurality of resources such as storage space, upload bandwidth, and download bandwidth are calculated and balanced based on previous resource prices, a supply of the resources, and a demand for the resources. Further, prices for operations such as storage and retrieval are determined such that a total of the payments to resource suppliers equals a total of the payments received from the resource consumers. In some embodiments, incoming data operation requests are allocated to the peers such that equilibrium among the peers is achieved.

Abstract: Distribution displays for categories are provided which illuminate the distribution of continuous attributes over all cases in a category, and which provide a histogram of the population of the different states of categorical attributes. An array of such displays by attribute (in one dimension) and category (in another dimension) may be provided. Category diagram displays are also provided for visualizing the different categories, and their distributions, populations, and similarities. These are displayed through different shading of nodes and edges representing categories and the relationship between two categories, and through proximity of nodes.

Abstract: Identification of email forwarders is described. In an implementation, a method includes using heuristics to identify email forwarders for use in a reputation system for locating spammers. In another implementation, a method includes determining a likelihood that a particular Internet Protocol (IP) address corresponds to an email forwarder and processing email originating from the particular IP address based on the determined likelihood. In a further implementation, a method includes collecting heuristic data that describes characteristics of emails sent from one or more Internet Protocol (IP) addresses and constructing a model from the heuristic data for identifying whether at least one of the IP address is an email forwarder. In yet a further implementation, a method includes identifying that a particular Internet Protocol (IP) address likely corresponds to an email forwarder and processing email from the particular IP address based on an implied sender of the email.