Abstract: An example system includes a processor to receive a test set, data slices, and a measure of interest. The processor can rank the data slices based on the test set, the data slices, and the set of measures of interest. The test set includes data points from the same feature space used to train a machine learning model. Each data slice is ranked according to generated slice grades representing unique information contribution of each data slice to the measure of interest with respect to the other data slices. The processor can then present the ranked data slices.

Abstract: A computer hardware system includes a slice generator and a policy generator and performs the following. The slice generator slices a first dataset including true values and predicted values of a class variable into a plurality of slices each defining a plurality of observations within the first dataset. A first one and another one of the plurality of slices are selected, and a union of observations is generated by adding observations within the selected another one to observations within the selected first one of the plurality of slices. The selecting another one of the plurality of slices and the generating the union is repeated until a number of observations within the union reaches a predetermined value. Using the policy generator and after the number of observations within the union reaches the predetermined value, an error policy is generated. The predicted values were generated by a machine learning engine.

Abstract: An example system includes a processor to receive a data set. The processor can generate a data slice rule based on a data observation for a data point in the data set. The processor can generate an instance of data based on the generated data slice rule.

Abstract: A computer-implemented method, according to one embodiment, includes: intentionally causing faults to be injected in a compute infrastructure, and determining whether the injected faults cause application failures. Weights are also assigned to the injected faults based on severity of the respective application failures. The weighted faults are compared, and changes to the compute infrastructure are recommended based on the comparison. Moreover, the changes that are recommended are configured to prevent the application failures. Other systems, methods, and computer program products are described in additional embodiments.

Abstract: Systems and techniques that facilitate ground-truth-less performance prediction of generative question-answering systems are provided. In various embodiments, a system can access a large language model (LLM) and a natural language question for which a ground-truth answer is unavailable. In various aspects, the system can generate, via a machine learning classifier that receives as input a set of properties associated with the natural language question, a classification label indicating whether or not the large language model will correctly answer the natural language question. In various instances, the set of properties can include a semantic category of the natural language question, a subject popularity of the natural language question, a semantic consistency exhibited by the LLM in response to repeated executions on the natural language question, or a semantic consistency exhibited by the LLM in response to execution on paraphrases of the natural language question.

Abstract: Various systems and methods are presented regarding detecting data drift. The data of interest can be batches of utterances received at an interface (e.g., a chatbot). The batches of utterances can be compared with topics present in training data utilized to train a data classifier (e.g., an autoencoder), wherein topics identified in the batches of utterances that are not present in the training data can be considered to be novel topics. The greater the presence of novel topics in a batch of utterances, the greater the divergence of the batch of utterances from the content of the training data. The novel topics can be identified and subsequently applied to the training data such that the data classifier can be re-trained with the novel topics, thereby causing the data classifier to be contemporaneous with the novel topics. In an embodiment, the utterances can be short streams of text, symbols, and suchlike.

Abstract: A computer-implemented method comprising: receiving a set of candidate trained machine learning models and a set of evaluation dimensions; generating risk scores for each of the candidate trained machine learning models over each of the evaluation dimensions; determining correlations between the evaluation dimensions based, at least in part, on the generated risk scores; and performing an optimization calculation to identify a subset of the set of candidate trained machine learning models, wherein each of the candidate trained machine learning models in the subset optimizes an overall risk measure over all of the evaluation dimensions, wherein the optimization calculation is based, at least in part, on the determined correlations.

Abstract: A computer-implemented method, according to one embodiment, includes: intentionally causing faults to be injected in a compute infrastructure, and determining whether the injected faults cause application failures. Weights are also assigned to the injected faults based on severity of the respective application failures. The weighted faults are compared, and changes to the compute infrastructure are recommended based on the comparison. Moreover, the changes that are recommended are configured to prevent the application failures. Other systems, methods, and computer program products are described in additional embodiments.

Abstract: Various systems and methods are presented regarding detecting data drift. The data of interest can be batches of utterances received at an interface (e.g., a chatbot). The batches of utterances can be compared with topics present in training data utilized to train a data classifier (e.g., an autoencoder), wherein topics identified in the batches of utterances that are not present in the training data can be considered to be novel topics. The greater the presence of novel topics in a batch of utterances, the greater the divergence of the batch of utterances from the content of the training data. The novel topics can be identified and subsequently applied to the training data such that the data classifier can be re-trained with the novel topics, thereby causing the data classifier to be contemporaneous with the novel topics. In an embodiment, the utterances can be short streams of text, symbols, and suchlike.

Abstract: A computer hardware system includes a slice generator and a policy generator and performs the following. The slice generator slices a first dataset including true values and predicted values of a class variable into a plurality of slices each defining a plurality of observations within the first dataset. A first one and another one of the plurality of slices are selected, and a union of observations is generated by adding observations within the selected another one to observations within the selected first one of the plurality of slices. The selecting another one of the plurality of slices and the generating the union is repeated until a number of observations within the union reaches a predetermined value. Using the policy generator and after the number of observations within the union reaches the predetermined value, an error policy is generated. The predicted values were generated by a machine learning engine.

Abstract: An example system includes a processor to receive a stream of records. The processor can generate an unbiased outlier score for each sample in the stream of records via a trained histogram-based outlier score model. The unbiased outlier score is unbiased for samples including dependent features using feature grouping. The processor can then detect an anomaly in response to detecting that an associated unbiased outlier score of the sample is higher than a predefined threshold.

Abstract: An example system includes a processor to receive a data set. The processor can generate a data slice rule based on a data observation for a data point in the data set. The processor can generate an instance of data based on the generated data slice rule.

Abstract: An example system includes a processor to receive a test set, data slices, and a measure of interest. The processor can rank the data slices based on the test set, the data slices, and the set of measures of interest. The test set includes data points from the same feature space used to train a machine learning model. Each data slice is ranked according to generated slice grades representing unique information contribution of each data slice to the measure of interest with respect to the other data slices. The processor can then present the ranked data slices.

Abstract: Systems and methods for automatically identifying in a dataset insufficient data for learning, or records with anomalous combinations of feature values, by partition of numeric and/or categorical data space into human-interpretable regions are disclosed. The method comprises: receiving a dataset of numeric and/or categorical features with a plurality of observations. Calculating observation density for each observation according to a distance or anomaly based metric, and receiving a density measurement. Partitioning the dataset along the numeric and/or categorical features according to the density measurement of each observation by a perpendicular cut along the feature spaces, receiving a map of a plurality of hyper-rectangular shapes representing various levels of density including empty spaces.

Abstract: A system, program product, and method for automatic detection of data drift in a data set are presented. The method includes determining changes to relations in the data set through generating baseline and production data sets. The method further includes generating a production data set with some inserted data distortion, and defining, for a plurality of features in the baseline data set, potential relations for participant features. The method also includes determining a first likelihood and a second likelihood of each potential relation in the baseline and production data sets, respectively, for the participant features. The method further includes comparing each first likelihood with each second likelihood, generating a comparison value that is compared with a threshold value, and determining, subject to the comparison value exceeding the threshold value, the potential relation in the baseline data set does not describe a relation in the production data set.

Abstract: A method, apparatus and product for reducing a number of test templates in a test suite. The method comprises determining, for a first test template of the test suite, a first probabilities vector comprising a first plurality of coverage probabilities with respect to a set of coverage events. The method comprises determining, for a second test template of the test suite, a second probabilities vector comprising a second plurality of coverage probabilities with respect to the set of coverage events. The method further comprises determining that the first test template is statistically dominant over the second test template based on the first probabilities vector and based on the second probabilities vector. The method further comprises providing an output based on the determination of the statistically dominant test template.

Abstract: A method, apparatus and product for reducing a number of test templates in a test suite. The method comprises determining, for a first test template of the test suite, a first probabilities vector comprising a first plurality of coverage probabilities with respect to a set of coverage events. The method comprises determining, for a second test template of the test suite, a second probabilities vector comprising a second plurality of coverage probabilities with respect to the set of coverage events. The method further comprises determining that the first test template is statistically dominant over the second test template based on the first probabilities vector and based on the second probabilities vector. The method further comprises providing an output based on the determination of the statistically dominant test template.