Abstract: A method, computer program product, and computer system for improving accuracy of results obtained from use of a generative artificial intelligence (GAI) model. The GAI model is executed to generate Q answers to Q questions Q prompts, respectively. A c-score is computed for each c-metric of multiple c-metrics of the Q questions. The computed c-score exceeds zero for X c-metrics of the multiple c-metrics. X Retrieval-Augmented Generation (RAG) shapes respectively corresponding to the X c-metrics are generated. Multiple RAG shape multiplets and associated multiplet scores are determined using the X RAG shapes. A top RAG shape multiplet having a highest RAG multiplet score is selected from the multiple RAG shape multiplets. The top RAG shape multiplet is graphically displayed on a display device. The GAI model's accuracy is improved after a root cause of the GAI model's inaccuracy was identified from the graphically displayed top RAG shape multiplet.

Abstract: An approach is provided in which the approach receives scored records that include a selected scored record comprising a first fairness group attribute, a first prediction, and a first confidence value corresponding to the first prediction. The approach perturbs the selected scored record to a second fairness group attribute in response to determining that the first confidence value is below a confidence threshold. The approach scores the perturbed record to generate a second prediction and a second confidence value corresponding to the second prediction. The approach modifies the selected scored record by changing the first prediction to the second prediction in response to determining that the second prediction is different from the first prediction and that the second confidence value is higher than the confidence threshold.

Abstract: One or more systems, devices, computer program products and/or computer-implemented methods of use provided herein relate to a process to facilitate abnormal document self-discovery. A system can comprise a memory that stores computer executable components, and a processor that executes the computer executable components stored in the memory, wherein the computer executable components can comprise a fairness component, an identification component, a removal component, and an evaluation component. The fairness component can receive a dataset for a tree-based model and calculates a first fairness, and the identification component can identify root to leaf paths in the tree-based model for one or more records of the dataset and one or more corresponding perturbed records. The removal component can remove at least one record of the one or more records having similar root to leaf paths to the other records of the one or more records.

Abstract: A machine learning model data quality improvement detection tool is provided for identifying an accurate reference group and an accurate monitored group of a machine learning model. The tool monitors a behavior of the machine learning model for a predetermined time frame. The tool compares a determined fairness metric a pre-defined fairness threshold. Responsive to the fairness metric failing to meet the pre-defined fairness threshold, the tool modifies the monitored group to include a first portion of the reference group. The tool compares a newly determined fairness metric to the pre-defined fairness threshold. Responsive to the newly determined fairness metric meeting the pre-defined fairness threshold, the tool identifies the modified monitored group including the first portion of the user-defined reference group as a new monitored group and the modified reference group without the first portion of the user-defined reference group as a new reference group.

Abstract: A computer-implemented method includes: receiving, by a computing device and from a user device, a request to validate an application program interface (API); validating, by the computing device, the API by performing a fetch analysis using different user profiles; returning, by the computing device and to the user device, a result of the fetch analysis; validating, by the computing device, the API by performing an insert/update analysis using the different user profiles; and returning, by the computing device and to the user device, a result of the insert/update analysis.

Abstract: An example operation may include one or more of storing a batch scoring engine and an application programming interface (API) for the batch scoring engine, receiving a trigger to perform a batch prediction process, reading input data from a source data store and executing, via the batch scoring engine, one or more predictive models on the input data to generate a predictive output and metadata associated with the predictive output, storing the predictive output and the metadata in a target data store, and updating the API with a location of the predictive output within the target data store and a location of the metadata within the target data store.

Abstract: Provided are techniques for correcting a classification model. For each original record of a plurality of original records that are processed by a classification model: the original record is perturbed; for the original record, an original confidence value is obtained for each class of a plurality of classes; for the perturbed record, a perturbed confidence value is obtained for each class of the plurality of classes; a final confidence value is determined using each original confidence value, each perturbed confidence value, and a direction of distance travelled; and a determination is made of whether the original record is biased based on the final confidence value. Then, it is determined whether the classification model is biased based on the original records that are determined to be biased. In response to determining that the classification model is biased, the classification model is corrected, otherwise, the classification model is deployed.

Abstract: A system and related method score a fairness of an outcome model. The method comprises receiving a set of original transaction records (OTRs), and selecting an OTR subset of the OTRs according to a subset selection criteria in order to reduce a number of OTRs to send to outcome model. For each OTR in the subset a perturbed transaction record (PTR) is created based on the OTR that includes changing at least one attribute in the PTR from the OTR, sending the OTR and the PTR to the outcome model, receiving an OTR outcome and a PTR outcome from the outcome model, and determining a record bias score for the OTR outcome and the PTR outcome respectively that indicates bias in the respective outcome. The OTR and the PTR bias score are stored in a bias determination system (BDS) database.

Abstract: Early indications of application programming interface (API) usage are identified by correlation to particular issues with the API including singular and mutual consistency, completeness, accuracy, and staleness. Analysis of API input and output along with data type and formatting information facilitates identification of the API issues. Establishing a correlation between API usage and issues supports early detection of potential usage reduction on a case-by-case level. Corrective action to resolve identified issues may be performed in a timely manner to maintain usage levels.

Abstract: One or more systems, devices, computer program products and/or computer-implemented methods of use provided herein relate to a process to facilitate abnormal document self-discovery. A system can comprise a memory that stores computer executable components, and a processor that executes the computer executable components stored in the memory, wherein the computer executable components can comprise a fairness component, an identification component, a removal component, and an evaluation component. The fairness component can receive a dataset for a tree-based model and calculates a first fairness, and the identification component can identify root to leaf paths in the tree-based model for one or more records of the dataset and one or more corresponding perturbed records. The removal component can remove at least one record of the one or more records having similar root to leaf paths to the other records of the one or more records.

Abstract: An example operation may include one or more of storing a batch scoring engine and an application programming interface (API) for the batch scoring engine, receiving a trigger to perform a batch prediction process, reading input data from a source data store and executing, via the batch scoring engine, one or more predictive models on the input data to generate a predictive output and metadata associated with the predictive output, storing the predictive output and the metadata in a target data store, and updating the API with a location of the predictive output within the target data store and a location of the metadata within the target data store.

Abstract: Early indications of application programming interface (API) usage are identified by correlation to particular issues with the API including singular and mutual consistency, completeness, accuracy, and staleness. Analysis of API input and output along with data type and formatting information facilitates identification of the API issues. Establishing a correlation between API usage and issues supports early detection of potential usage reduction on a case-by-case level. Corrective action to resolve identified issues may be performed in a timely manner to maintain usage levels.

Abstract: One example of a system comprises using a processor for identifying a model to be validated that is stored in a repository; automatically computing and recording one or more model metrics for the model to be validated in a tamper-proof manner; comparing the computed tamper-proof metrics with one or more encoded rules and policies to determine if the model to be validated complies with the one or more encoded rules and policies; and outputting a notification to a device indicating a validation status of the model to be validated based on the comparison of the computed tamper-proof metrics with the one or more encoded rules and policies.

Abstract: Artificial intelligence (AI) model monitoring and ranking includes obtaining metric values indicative of performance of AI model deployments, the metric values including respective metric values measured across metrics, determining violation statuses of the metrics for each of the AI model deployments, the violation statuses indicating, for each AI model deployment, which of the metrics are violated by the AI model deployment as reflected by respective metric values for that AI model deployment, ranking the AI model deployments against each other according to a ranking model and based on the determined violation statuses for each of the AI model deployments, and providing a rank of at least some of the AI model deployments to a user.

Abstract: A processor may receive an original dataset. The processor may segment, automatically, the original dataset into a plurality of data groups. The plurality of data groups may include a model training dataset and a holdout dataset. The processor may generate a model with the model training dataset. The processor may validate the model with the holdout dataset.

Abstract: One example of a system comprises using a processor for identifying a model to be validated that is stored in a repository; automatically computing and recording one or more model metrics for the model to be validated in a tamper-proof manner; comparing the computed tamper-proof metrics with one or more encoded rules and policies to determine if the model to be validated complies with the one or more encoded rules and policies; and outputting a notification to a device indicating a validation status of the model to be validated based on the comparison of the computed tamper-proof metrics with the one or more encoded rules and policies.

Abstract: A system includes a memory having instructions therein and at least one processor in communication with the memory. The at least one processor is configured to execute the instructions to run a machine learning base model on input data to generate base model prediction data and run a machine learning error prediction model on the input data to generate error prediction data. The at least one processor is configured to execute the instructions to generate predicted correct base model prediction data based on the base model prediction data and the error prediction data. The at least one processor is configured to execute the instructions to generate confusion values data based on the base model prediction data and the predicted correct base model prediction data. The at least one processor is also configured to execute the instructions to generate base model accuracy fairness metrics data based on the confusion values data.

Abstract: One example of a method comprises identifying a model to be validated that is stored in a repository; automatically computing and recording one or more model metrics for the model to be validated in a tamper-proof manner; comparing the computed tamper-proof metrics with one or more encoded rules and policies to determine if the model to be validated complies with the one or more encoded rules and policies; and outputting a notification to a device indicating a validation status of the model to be validated based on the comparison of the computed tamper-proof metrics with the one or more encoded rules and policies.

Abstract: Artificial intelligence (AI) model monitoring and ranking includes obtaining metric values indicative of performance of AI model deployments, the metric values including respective metric values measured across metrics, determining violation statuses of the metrics for each of the AI model deployments, the violation statuses indicating, for each AI model deployment, which of the metrics are violated by the AI model deployment as reflected by respective metric values for that AI model deployment, ranking the AI model deployments against each other according to a ranking model and based on the determined violation statuses for each of the AI model deployments, and providing a rank of at least some of the AI model deployments to a user.

Abstract: Disclosed embodiments provide techniques for monitoring application program interface (API) health based on design-time checks and runtime performance. The health of an API can include multiple factors. The factors may include correctness of the data returned/output by the API. A high amount of incorrect data may correlate to a lower health metric. A low amount of incorrect data may correlate to a higher health metric. If a health metric value falls below a predetermined threshold, it may indicate that the API health is poor, and hence, should not be used, or should be used with caution, depending on the application. In embodiments, an alternate API may be automatically selected as a replacement for an API in response to a health metric falling below a predetermined threshold.