Abstract: In an embodiment, a process for automatic model monitoring for data streams includes receiving an input dataset, using a machine learning model to determine a model score for each data record of at least a portion of the input dataset, and determining monitoring values. Each monitoring value is associated with a measure of similarity between model scores for those data records of the input dataset within a corresponding moving reference window and model scores for those data records of the input dataset within a corresponding moving target window. The process includes outputting the determined monitoring values.

Abstract: In an embodiment, a process for automatic model monitoring for data streams includes receiving an input dataset, using a machine learning model to determine a model score for each data record of at least a portion of the input dataset, and determining monitoring values. Each monitoring value is associated with a measure of similarity between model scores for those data records of the input dataset within a corresponding moving reference window and model scores for those data records of the input dataset within a corresponding moving target window. The process includes outputting the determined monitoring values.

Abstract: In an embodiment, a process for adaptive threshold estimation for streaming data includes determining initial positions for a set of percentile bins, receiving a new data item in a stream of data, and identifying one of the set of percentile bins corresponding to the new data item. The process includes incrementing a count of items in the identified percentile bin, adjusting one or more counts of data items in one or more of the percentile bins including by applying a suppression factor based on a relative ordering of items, and redistributing positions for the set of percentile bins to equalize respective count numbers of items for each percentile bin of the set of percentile bins. The process includes utilizing the redistributed positions of the set of percentile bins to determine a percentile distribution of the data stream, and calculating a threshold based at least in part on the percentiles distribution.

Abstract: In an embodiment, a process for automatic model monitoring for data streams includes receiving an input dataset, using a machine learning model to determine a model score for each data record of at least a portion of the input dataset, and determining monitoring values. Each monitoring value is associated with a measure of similarity between model scores for those data records of the input dataset within a corresponding moving reference window and model scores for those data records of the input dataset within a corresponding moving target window. The process includes outputting the determined monitoring values.

Abstract: A data stream is received. Data elements of the data stream are analyzed using one or more machine learning models and one or more machine learning prediction explanation implementations. Different candidate presentations are tested. The different candidate presentations are associated with machine learning results provided to different reviewers in a group of human-in-the-loop reviewers that review predictions of the one or more machine learning models. The different candidate presentations include different explanations generated by the one or more machine learning prediction explanation implementations and at least one control candidate presentation corresponding to an absent explanation. Different aspects of the testing are monitored. Results of the monitoring are used to make a selection among the different candidate presentations.

Abstract: In an embodiment, a process for automatic model monitoring for data streams includes receiving an input dataset, using a machine learning model to determine a model score for each data record of at least a portion of the input dataset, and determining monitoring values. Each monitoring value is associated with a measure of similarity between model scores for those data records of the input dataset within a corresponding moving reference window and model scores for those data records of the input dataset within a corresponding moving target window. The process includes outputting the determined monitoring values.

Abstract: In an embodiment, a process for adaptive threshold estimation for streaming data includes determining initial positions for a set of percentile bins, receiving a new data item in a stream of data, and identifying one of the set of percentile bins corresponding to the new data item. The process includes incrementing a count of items in the identified percentile bin, adjusting one or more counts of data items in one or more of the percentile bins including by applying a suppression factor based on a relative ordering of items, and redistributing positions for the set of percentile bins to equalize respective count numbers of items for each percentile bin of the set of percentile bins. The process includes utilizing the redistributed positions of the set of percentile bins to determine a percentile distribution of the data stream, and calculating a threshold based at least in part on the percentiles distribution.

Abstract: In an embodiment, a process for explanation reporting based on differentiation between items in different data groups includes obtaining model scores from a first machine learning model and training a second machine learning model to learn how to differentiate between two groups based on at least one of: features and the model scores obtained from the first machine learning model. The process includes applying the second machine learning model to each data record in a first group of data records to determine a corresponding ranking score for each data record in the first group, and based on the corresponding ranking scores, determining a relative contribution of each of the data records in the first group to the differentiation between the first group of data records and a second group of data records.