Abstract: Generally discussed herein are devices, systems, and methods for generating a phrase that is confusing to a language classifier. A method can include determining, by the LC, a first classification score (CS) of a prompt indicating whether the prompt is a first class or a second class, predicting, based on the prompt and by a pre-trained language model (PLM), likely next words and a corresponding probability for each of the likely next words, determining, by the LC, a second CS for each of the likely next words, determining, by an adversarial classifier, respective scores for each of the likely next words, the respective scores determined based on the first CS of the prompt, the second CS of the likely next words, and the probabilities of the likely next words, and selecting, by an adversarial classifier, a next word of the likely next words based on the respective scores.

Abstract: The disclosure relates to a sensitivity detection system that accurately and efficiently determines when information based on a user's browsing activity unintentionally reveals private or other sensitive information about the user. For example, the sensitivity detection system generates and utilizes machine learning models for detecting sensitivity to accurately detect when sensitive user information is being leaked from a collection of user information, such as a user profile. Additionally, upon determining that sensitive user information is being revealed, in many instances, the sensitivity detection system performs mitigation actions to stop and/or reduce sensitive user information from being undesirably revealed.

Abstract: Generally discussed herein are devices, systems, and methods for generating a phrase that is confusing to a language classifier. A method can include determining, by the LC, a first classification score (CS) of a prompt indicating whether the prompt is a first class or a second class, predicting, based on the prompt and by a pre-trained language model (PLM), likely next words and a corresponding probability for each of the likely next words, determining, by the LC, a second CS for each of the likely next words, determining, by an adversarial classifier, respective scores for each of the likely next words, the respective scores determined based on the first CS of the prompt, the second CS of the likely next words, and the probabilities of the likely next words, and selecting, by an adversarial classifier, a next word of the likely next words based on the respective scores.

Abstract: The present disclosure relates to systems, methods, and computer readable media that evaluate performance of a machine learning system in connection with a test dataset. For example, systems disclosed herein may receive a test dataset and identify label information for the test dataset including feature information and ground truth data. The systems disclosed herein can compare the ground truth data and outputs generated by a machine learning system to evaluate performance of the machine learning system with respect to the test dataset. The systems disclosed herein may further generate feature clusters based on failed outputs and corresponding features and generate a number of performance views that illustrate performance of the machine learning system with respect to clustered groupings of the test dataset.

Abstract: Generally discussed herein are devices, systems, and methods for generating a phrase that is confusing to a language classifier. A method can include determining, by the LC, a first classification score (CS) of a prompt indicating whether the prompt is a first class or a second class, predicting, based on the prompt and by a pre-trained language model (PLM), likely next words and a corresponding probability for each of the likely next words, determining, by the LC, a second CS for each of the likely next words, determining, by an adversarial classifier, respective scores for each of the likely next words, the respective scores determined based on the first CS of the prompt, the second CS of the likely next words, and the probabilities of the likely next words, and selecting, by an adversarial classifier, a next word of the likely next words based on the respective scores.

Abstract: The present disclosure relates to systems, methods, and computer readable media that evaluate performance of a machine learning system in connection with a test dataset. For example, systems disclosed herein may receive a test dataset and identify label information for the test dataset including feature information and ground truth data. The systems disclosed herein can compare the ground truth data and outputs generated by a machine learning system to evaluate performance of the machine learning system with respect to the test dataset. The systems disclosed herein may further generate feature clusters based on failed outputs and corresponding features and generate a number of performance views that illustrate performance of the machine learning system with respect to clustered groupings of the test dataset.

Abstract: The present disclosure relates to systems, methods, and computer readable media that evaluate performance of a machine learning system in connection with a test dataset. For example, systems disclosed herein may receive a test dataset and identify label information for the test dataset including feature information and ground truth data. The systems disclosed herein can compare the ground truth data and outputs generated by a machine learning system to evaluate performance of the machine learning system with respect to the test dataset. The systems disclosed herein may further generate feature clusters based on failed outputs and corresponding features and generate a number of performance views that illustrate performance of the machine learning system with respect to clustered groupings of the test dataset.

Abstract: The present disclosure relates to systems, methods and computer readable media for evaluating performance of a machine learning system and providing one or more performance views representative of the determined performance. For example, systems disclosed herein may receive or identify performance information including outputs, accuracy data, and feature data associated with a plurality of test instances. In addition, systems disclosed herein may provide one or more performance views via a graphical user interface including graphical elements (e.g., interactive elements) and indications of accuracy data and other performance data with respect to feature clusters associated with select groupings of test instances from the plurality of test instances. The performance views may include interactive features to enable a user to view and intuitively understand performance of the machine learning system with respect to clustered groupings of test instances that share common characteristics.

Abstract: Edits on a content item, such as a document, are divided into microtasks. The microtasks associated with a document can be automatically identified based on a workflow or can be identified by a user associated with the content item or an administrator. At a later time, the user can complete the microtasks for a content item using an application associated with their smart phone or tablet. The application may present the microtasks in a game-like environment where the user can compete with other users based on metrics such as number of microtasks completed in a day or fastest completion time. In addition, the user can earn rewards such as badges, coupons, or credits by completing microtasks. In this way, users can use time that would have been wasted playing games to complete their content items, while still experiencing some of the fun and competition associated with the games.

Abstract: Edits on a content item, such as a document, are divided into microtasks. The microtasks associated with a document can be automatically identified based on a workflow or can be identified by a user associated with the content item or an administrator. At a later time, the user can complete the microtasks for a content item using an application associated with their smart phone or tablet. The application may present the microtasks in a game-like environment where the user can compete with other users based on metrics such as number of microtasks completed in a day or fastest completion time. In addition, the user can earn rewards such as badges, coupons, or credits by completing microtasks. In this way, users can use time that would have been wasted playing games to complete their content items, while still experiencing some of the fun and competition associated with the games.

Abstract: A stochastic privacy service provider may provide users with a guaranteed upper bound on a probability that personal data will be accessed while enabling the services to collect data that can be used to enhance its services. Users may receive incentives to become participants in a stochastic privacy program. The stochastic privacy provider may employ one or more probabilistic and decision-theoretic methods to determine which participants' personal data should be sought while guaranteeing that the probability of personal data being accessed is smaller than the mutually agreed upon probability of access. The probability of access may be on a per time basis. The stochastic privacy provider may access coalescenses of the personal data of sets of multiple people, where a maximum probability is given for accessing statistical summaries of personal data computed from groups of people that are of at least some guaranteed size.

Abstract: Various technologies pertaining to dynamically identifying travel segments to be taken by a traveler traveling in a region are described herein, where observations about travel segments in the region are sparse and subject to alteration. A computer-implemented graph can be loaded into a memory, where the computer-implemented graph is representative of the region. The computer-implemented graph includes nodes that represent locations in the region and edges that represent travel segments of the region, where the edges have costs assigned thereto, and further where there is a defined statistical relationship between the costs. When an observation about a travel path is received, using the computer-implemented graph, inferences can be made about costs of traversing other travel paths in the region.

Abstract: Edits on a content item, such as a document, are divided into microtasks. The microtasks associated with a document can be automatically identified based on a workflow or can be identified by a user such as the creator of the content item or an administrator. The microtasks can be assigned to one or more workers including the creator of the content item. When a determination is made that an assigned worker is available to complete a microtask (e.g., when the worker is waiting in line, has just closed an application or file, or has just completed a phone call, etc.), the assigned microtask is presented to the worker for completion.

Abstract: Edits on a content item, such as a document, are divided into microtasks. The microtasks associated with a document can be automatically identified based on a workflow or can be identified by a user associated with the content item or an administrator. At a later time, the user can complete the microtasks for a content item using an application associated with their smart phone or tablet. The application may present the microtasks in a game-like environment where the user can compete with other users based on metrics such as number of microtasks completed in a day or fastest completion time. In addition, the user can earn rewards such as badges, coupons, or credits by completing microtasks. In this way, users can use time that would have been wasted playing games to complete their content items, while still experiencing some of the fun and competition associated with the games.

Abstract: Various technologies pertaining to dynamically identifying travel segments to be taken by a traveler traveling in a region are described herein, where observations about travel segments in the region are sparse and subject to alteration. A computer-implemented graph can be loaded into a memory, where the computer-implemented graph is representative of the region. The computer-implemented graph includes nodes that represent locations in the region and edges that represent travel segments of the region, where the edges have costs assigned thereto, and further where there is a defined statistical relationship between the costs. When an observation about a travel path is received, using the computer-implemented graph, inferences can be made about costs of traversing other travel paths in the region.

Abstract: A crowdsourcing environment is described herein which uses a single-stage or multi-stage approach to evaluate the quality of work performed by a worker, with respect to an identified task. In the multi-stage case, an evaluation system, in the first stage, determines whether the worker corresponds to a spam agent. In a second stage, for a non-spam worker, the evaluation system determines the propensity of the worker to perform desirable (e.g., accurate) work in the future. The evaluation system operates based on a set of features, including worker-focused features (which describe work performed by the particular worker), task-focused features (which describe tasks performed in the crowdsourcing environment), and system-focused features (which describe aspects of the configuration of the crowdsourcing environment). According to one illustrative aspect, the evaluation system performs its analysis using at least one model, produced using any type of supervised machine learning technique.

Abstract: Various embodiments are described for computerized advertising systems and methods. The system may include an ad server that includes an impression campaign engine configured to associate a target user profile with a plurality of computing devices. The ad server is also configured to receive a multi-step impression plan including a plurality of triggers from an advertiser. Each trigger is associated with a different advertisement to be served to at least one of the plurality of devices. The system also includes an ad serving engine configured to serve a first advertisement to a first device in response to making an inference from sensors or detecting a first trigger, and a second advertisement to a second device in response to a second inference or detecting a second trigger, according to the impression plan. A predictive model developed from machine learning may be used to develop a learning-based multi-step impression plan.

Abstract: The claimed matter provides systems and/or techniques that develop or use predictive models of human forgetting to effectuate automated reminding. The system includes the use of predictive models that infer the probability that aspects of items will be forgotten, models that evaluate the relevance of recalling aspects of items in different settings, based on contextual information related to user attributes associated with the items, and models of the context-sensitive cost of interrupting users with reminders. The system can combine the probability of users forgetting aspects of an item with an assessed cost of forgetting those aspects to ascertain expected costs for not being reminded about events, compare expected costs for not being reminded with expected costs for interrupting users, and based on comparisons between expected costs for being reminded and expected costs for interrupting users regarding events, generate and deliver reminder notifications to users about items.

Abstract: The claimed matter provides systems and/or techniques that develop or use predictive models of human forgetting to effectuate automated reminding. The system includes the use of predictive models that infer the probability that aspects of items will be forgotten, models that evaluate the relevance of recalling aspects of items in different settings, based on contextual information related to user attributes associated with the items, and models of the context-sensitive cost of interrupting users with reminders. The system can combine the probability of users forgetting aspects of an item with an assessed cost of forgetting those aspects to ascertain expected costs for not being reminded about events, compare expected costs for not being reminded with expected costs for interrupting users, and based on comparisons between expected costs for being reminded and expected costs for interrupting users regarding events, generate and deliver reminder notifications to users about items.

Abstract: The claimed matter provides systems and/or techniques that develop or use predictive models of human forgetting to effectuate automated reminding. The system includes the use of predictive models that infer the probability that aspects of items will be forgotten, models that evaluate the relevance of recalling aspects of items in different settings, based on contextual information related to user attributes associated with the items, and models of the context-sensitive cost of interrupting users with reminders. The system can combine the probability of users forgetting aspects of an item with an assessed cost of forgetting those aspects to ascertain expected costs for not being reminded about events, compare expected costs for not being reminded with expected costs for interrupting users, and based on comparisons between expected costs for being reminded and expected costs for interrupting users regarding events, generate and deliver reminder notifications to users about items.