Abstract: In an embodiment, a process for automatic model monitoring for data streams includes receiving an input dataset, using a machine learning model to determine a model score for each data record of at least a portion of the input dataset, and determining monitoring values. Each monitoring value is associated with a measure of similarity between model scores for those data records of the input dataset within a corresponding moving reference window and model scores for those data records of the input dataset within a corresponding moving target window. The process includes outputting the determined monitoring values.

Abstract: In various embodiments, a process for automatic profile extraction in data streams using recurrent neural networks includes receiving input sequence data associated with a stream of events and using a plurality of trained recurrent neural network machine learning models at least in part in parallel to determine different embedding output sets that represent at least a portion of the input sequence data in a plurality of different embedding spaces. The process includes providing the different embedding output sets to one or more classifier machine learning models to determine one or more classifier results, and using the one or more classifier results to provide a prediction output.

Abstract: In an embodiment, a process for automated rules management system includes receiving a specification of past predicted results of evaluation rules and corresponding observed outcomes. The process includes determining one or more sets of alternative activations or priorities of at least a portion of the evaluation rules, assessing the one or more sets of alternative activations or priorities of at least a portion of the evaluation rules, and optimizing result activations or priorities of at least a portion of the evaluation rules based at least in part on the assessment of the one or more sets of alternative activations or priorities.

Abstract: A multi-task hierarchical machine learning model is configured to perform both a decision task to predict a decision result and an explanation task to predict a plurality of semantic concepts for explainability associated with the decision task, wherein a semantic layer of the machine learning model associated with the explanation task is utilized as an input to a subsequent decision layer of the machine learning model associated with the decision task. Training data is received. The multi-task hierarchical machine learning model is trained using the received training data.

Abstract: One or more events of a data stream are received. For each feature of a set of features, the one or more events are used to update a corresponding distribution of data from the data stream. For each feature of the set of features, the corresponding updated distribution and a corresponding reference distribution are used to determine a corresponding divergence value. For each feature of the set of features, the corresponding determined divergence value and a corresponding distribution of divergences are used to determine a corresponding statistical value. Using the statistical values each corresponding to a different feature of the set of features, a statistical analysis is performed to determine a result associated with a likelihood of data drift detection.

Abstract: A set of data elements is received. For each feature of a set of features, a corresponding reference distribution for the set of data elements is determined. For each feature of the set of features, one or more corresponding subset distributions for one or more subsets sampled from the set of data elements are determined. For each feature of the set of features, the corresponding reference distribution is compared with each of the one or more corresponding subset distributions to determine a corresponding distribution of divergences. At least the determined distributions of divergences for the set of features are provided for use in automated data analysis.

Abstract: A graph of nodes and edges is received. An identification of a starting node in the graph is received. Traversal walks on the graph from the starting node are automatically performed, wherein performing each of the traversal walks includes traversing to a randomly selected next node until any of one or more stopping criteria is met. One or more processors are used to determine one or more metrics based on the traversal walks. At least a portion of the one or more metrics is used to predict an illicit activity or entity.

Abstract: In various embodiments, a process for providing a distributed streaming system supporting real-time sliding windows includes receiving a stream of events at a plurality of distributed nodes and routing the events into topic groupings. The process includes using one or more events in at least one of the topic groupings to determine one or more metrics of events with at least one window and an event reservoir including by: tracking, in a volatile memory of the event reservoir, beginning and ending events within the at least one window; and tracking, in a persistent storage of the event reservoir, events associated with tasks assigned to a respective node. The process includes updating the one or more metrics based on one or more previous values of the one or more metrics as a new event is added or an existing event is expired from the at least one window.

Abstract: A series of sequential inputs and a prediction output of a machine learning model, to be analyzed for interpreting the prediction output, are received. An input included in the series of sequential inputs is selected to be analyzed for relevance in producing the prediction output. Background data for the selected input of the series of sequential inputs to be analyzed is determined. The background data is used as a replacement for the selected input of the series of sequential inputs to determine a plurality of perturbed prediction outputs of the machine learning model. A relevance metric is determined for the selected input based at least in part on the plurality of perturbed prediction outputs of the machine learning model.

Abstract: A multi-task hierarchical machine learning model is configured to perform both a decision task to predict a decision result and an explanation task to predict a plurality of semantic concepts for explainability associated with the decision task, wherein a semantic layer of the machine learning model associated with the explanation task is utilized as an input to a subsequent decision layer of the machine learning model associated with the decision task. Training data is received. The multi-task hierarchical machine learning model is trained using the received training data.

Abstract: A labeling function associated with generating one or more semantic concepts is received. The received labeling function is used to automatically annotate an existing dataset with the one or more semantic concepts to generate an annotated noisy dataset. A reference dataset annotated with the one or more semantic concepts is received. A training dataset is prepared including by combining at least a portion of the reference dataset with at least a portion of the annotated noisy dataset. The training dataset is used to train a multi-task machine learning model configured to perform both a decision task to predict a decision result and an explanation task to predict a plurality of semantic concepts for explainability associated with the decision task.

Abstract: Input data is received. The received input data is provided to a trained discriminative machine learning model to determine an inference result. At least a portion of the received input data is used to determine a utility measure. A version of the determined inference result and the utility measure are used as inputs to a decision module optimizing one or more decision metrics to determine a decision result.

Abstract: In various embodiments, a process for providing a distributed streaming system supporting real-time sliding windows includes receiving a stream of events at a plurality of distributed nodes and routing the events into topic groupings. The process includes using one or more events in at least one of the topic groupings to determine one or more metrics of events with at least one window and an event reservoir including by: tracking, in a volatile memory of the event reservoir, beginning and ending events within the at least one window; and tracking, in a persistent storage of the event reservoir, all events associated with tasks assigned to a respective node. The process includes updating the one or more metrics based on one or more previous values of the one or more metrics as a new event is added or an existing event is expired from the at least one window.

Abstract: In various embodiments, a process for providing a distributed streaming system supporting real-time sliding windows includes receiving a stream of events at a plurality of distributed nodes and routing the events into topic groupings. The process includes using one or more events in at least one of the topic groupings to determine one or more metrics of events with at least one window and an event reservoir including by: tracking, in a volatile memory of the event reservoir, beginning and ending events within the at least one window; and tracking, in a persistent storage of the event reservoir, all events associated with tasks assigned to a respective node. The process includes updating the one or more metrics based on one or more previous values of the one or more metrics as a new event is added or an existing event is expired from the at least one window.

Abstract: A data stream is received. Data elements of the data stream are analyzed using one or more machine learning models and one or more machine learning prediction explanation implementations. Different candidate presentations are tested. The different candidate presentations are associated with machine learning results provided to different reviewers in a group of human-in-the-loop reviewers that review predictions of the one or more machine learning models. The different candidate presentations include different explanations generated by the one or more machine learning prediction explanation implementations and at least one control candidate presentation corresponding to an absent explanation. Different aspects of the testing are monitored. Results of the monitoring are used to make a selection among the different candidate presentations.

Abstract: In various embodiments, a process for fairness-aware hyperparameter optimization based on bandit-based techniques includes receiving a fairness evaluation metric for evaluating a fairness of a machine learning model to be trained and receiving a performance metric for evaluating performance of the machine learning model to be trained. The process includes automatically evaluating candidate combinations of hyperparameters of the machine learning model based at least in part on multi-objective optimization including scalarization and using the fairness evaluation metric and the performance metric to select a hyperparameter combination to utilize among the candidate combinations of hyperparameters, wherein evaluating the candidate combinations of hyperparameters of the machine learning model includes automatically and dynamically determining a relative weighting between the fairness evaluation metric and the performance metric.

Abstract: In various embodiments, a process for providing an active learning annotation system that does not require historical data includes receiving a stream of unlabeled data, identifying a portion of the unlabeled data to label without access to label information, and receiving a labeled version of the identified portion of the unlabeled data and storing the labeled version as labeled data. The process includes analyzing the labeled version and at least a portion of the received unlabeled data that has not been labeled to identify an additional portion of the unlabeled data to label and store in the labeled data including by applying at least one warm up policy.

Abstract: A process for handling interleaved sequences using RNNs includes receiving data of a first transaction, retrieving a first state (e.g., a default or a saved RNN state for an entity associated with the first transaction), and determining a new second state and a prediction result using the first state and an input data based on the first transaction. The process includes updating the saved RNN state for the entity to be the second state. The process includes receiving data of a second transaction, where the second transaction is associated with the same entity as the first transaction. The process unloops an RNN associated with the saved RNN state including by: retrieving the second state, determining a new third state and a prediction result using the second state and an input data based the second transaction, and updating the saved RNN state for the entity to be the third state.

Abstract: Computer memory management during real-time fraudulent transaction analysis is disclosed. In an embodiment, a method includes receiving a transaction entry and determining whether the transaction entry is potentially fraudulent, including by determining whether the transaction entry is correlated with another transaction entry in the computer memory or an already-identified suspicious pattern of other transaction entries. The method further includes determining whether to evict the transaction entry from the computer memory based on the determination of whether the transaction entry is potentially fraudulent. The method includes providing a result of fraudulent transaction analysis performed using the computer memory that has been optimized.

Abstract: In an embodiment, a process for automated rules management system includes receiving a specification of past predicted results of evaluation rules and corresponding observed outcomes. The process includes determining one or more sets of alternative activations or priorities of at least a portion of the evaluation rules, assessing the one or more sets of alternative activations or priorities of at least a portion of the evaluation rules, and optimizing result activations or priorities of at least a portion of the evaluation rules based at least in part on the assessment of the one or more sets of alternative activations or priorities.