Abstract: Techniques are disclosed herein for implementing planning mechanisms for agentic artificial intelligence (AI) systems. A query concerning a person and a request for a context identifier is received from a client device. A data set describing characteristics for a set of persons is accessed. The context identifier is generated and transferred to the client device. An event message including an event associated with the query and the context identifier is received from the client device. A determination is made that a characteristic for the person is included in characteristics and an input for a generative machine learning model is generated. An execution plan for generating a response to the query is obtained using the generative machine learning model and the input and a response to the query is generated by executing the execution plan.

Abstract: Techniques disclosed herein pertain to agentic artificial intelligence (AI) systems, and, more specifically, to safety mechanisms for agentic AI systems. A query is accessed, and, in response to accessing the query, a planning process to generate an execution plan for responding to the query and a safety analysis process to determine a safety classification for the query based on a set of policies can be initiated. A determination can be made as to whether to authorize execution of the execution plan based on the safety classification from the safety analysis process. In response to determining to authorize execution of the execution plan, the execution plan can be executed and a substantive response to the query can be generated based on the execution. In response to determining not to authorize execution of the execution plan, execution of the execution plan can be blocked and a fallback response can be generated.

Abstract: Views may be generated for bias metrics or feature attribution captured in machine learning pipelines. A request to create a view of bias metrics or feature attribution may be received. The bias metrics or feature attribution may have been determined in a machine learning pipeline as part of executing a training job that specified the bias metrics or the feature attribution. A development application may access a data store that stores the bias metrics or the feature attribution determined in the machine learning pipeline. A view based on the bias metrics or feature attribution may be generated and provided.

Abstract: A determination is made that an explanatory data set for a common set of predictions generated by a machine learning model for records containing text tokens is to be provided. Respective groups of related tokens are identified from the text attributes of the records, and record-level prediction influence scores are generated for the token groups. An aggregate prediction influence score is generated for at least some of the token groups from the record-level scores, and an explanatory data set based on the aggregate scores is presented.

Abstract: Systems and methods for implementing auditing of large language model-based tools for bias in inferences is disclosed. Individual entries of the dataset of dialogs may be modified to include stereotypical details of particular contexts. These modified records may then be submitted to an automated response generator to produce a set benchmark records. The baseline records and benchmark records may then be analyzed for completeness, accuracy and conciseness with respect to the particular contexts and disparities in precision and recall may be determined using differences in the benchmark and baseline records. The determined disparities may then be used to further train or fine-tune the automated response generator.

Abstract: Views may be generated for bias metrics or feature attribution captured in machine learning pipelines. A request to create a view of bias metrics or feature attribution may be received. The bias metrics or feature attribution may have been determined in a machine learning pipeline as part of executing a training job that specified the bias metrics or the feature attribution. A development application may access a data store that stores the bias metrics or the feature attribution determined in the machine learning pipeline. A view based on the bias metrics or feature attribution may be generated and provided.

Abstract: Bias metrics may be captured at different stages for training a machine learning model. A training job may specify bias metrics to capture at multiple different stages of a machine learning pipeline for a feature of a training data set used to train a machine learning model. The training job may be executed and the bias metrics determined at the stages as specified in the training job. The bias metrics for the different stages may be stored.

Abstract: Automatic failure diagnosis and correction may be performed on trained machine learning models. Input data that causes a trained machine learning model may be identified in order to determine different model failures. The model failures may be clustered in order to determine failure scenarios for the trained machine learning model. Examples of the failure scenarios may be generated and truth labels for the example scenarios obtained. The examples and truth labels may then be used to retrain the machine learning model to generate a corrected version of the machine learning model.

Abstract: In an embodiment, a method includes receiving, via a processor of a first compute device, a representation of a set of inputs and a set of outputs that were generated by inputting the set of inputs into a machine learning (ML) model by a set of compute devices not including the first compute device to generate the set of outputs. The method further includes receiving, via the processor, a request for a machine learning (ML) explanation associated with the ML model and at least one explicand. The method further includes generating, via the processor and without using the ML model, a representation of the ML explanation based on the at least one explicand, the set of inputs, and the set of outputs.

Abstract: Explanation jobs may be performed for computer vision tasks. A request to execute an explanation job for a computer vision machine learning model may be received. The execution job may be performed, including extracting different features from the image, determining the respective relative importance values of the different features on inferences generated by the computer vision machine learning model. The result of the explanation job, including the generated heat maps may be provided.

Abstract: A determination is made that an explanatory data set for a common set of predictions generated by a machine learning model for records containing text tokens is to be provided. Respective groups of related tokens are identified from the text attributes of the records, and record-level prediction influence scores are generated for the token groups. An aggregate prediction influence score is generated for at least some of the token groups from the record-level scores, and an explanatory data set based on the aggregate scores is presented.

Abstract: A determination is made that an explanatory data set for a common set of predictions generated by a machine learning model for records containing text tokens is to be provided. Respective groups of related tokens are identified from the text attributes of the records, and record-level prediction influence scores are generated for the token groups. An aggregate prediction influence score is generated for at least some of the token groups from the record-level scores, and an explanatory data set based on the aggregate scores is presented.

Abstract: An algorithm releases answers to very large numbers of statistical queries, e.g., k-way marginals, subject to differential privacy. The algorithm answers queries on a private dataset using simple perturbation, and then attempts to find a synthetic dataset that most closely matches the noisy answers. The algorithm uses a continuous relaxation of the synthetic dataset domain which makes the projection loss differentiable, and allows the use of efficient machine learning optimization techniques and tooling. Rather than answering all queries up front, the algorithm makes judicious use of a privacy budget by iteratively and adaptively finding queries for which relaxed synthetic data has high error, and then repeating the projection. The algorithm is effective across a range of parameters and datasets, especially when a privacy budget is small or a query class is large.

Abstract: A set of clusters from a first set of vector representations (VRs) is identified. A center associated with each cluster from the set of clusters to generate a set of centers is determined. For each VR from the first set of VRs, and to generate a first set of distributions, a distribution of that VR is determined that indicates, for each center from the set of centers, similarity between that VR and that center. For each VR from a second set of VRs, and to generate a second set of distributions, a distribution of that VR is determined that indicates, for each center from the set of centers, similarity between that VR and that center. A set of divergence metrics associated with the first set of VRs and the second set of VRs are computed based on comparing the first set of distributions and the second set of distributions.

Abstract: An algorithm releases answers to very large numbers of statistical queries, e.g., k-way marginals, subject to differential privacy. The algorithm answers queries on a private dataset using simple perturbation, and then attempts to find a synthetic dataset that most closely matches the noisy answers. The algorithm uses a continuous relaxation of the synthetic dataset domain which makes the projection loss differentiable, and allows the use of efficient machine learning optimization techniques and tooling. Rather than answering all queries up front, the algorithm makes judicious use of a privacy budget by iteratively and adaptively finding queries for which relaxed synthetic data has high error, and then repeating the projection. The algorithm is effective across a range of parameters and datasets, especially when a privacy budget is small or a query class is large.

Abstract: Hyperparameters for tuning a machine learning system may be optimized for fairness using Bayesian optimization with constraints for accuracy and bias. Hyperparameter optimization may be performed for a received training set and received accuracy and fairness constraints. Respective probabilistic models for accuracy and bias of the machine learning system may be initialized, then hyperparameter optimization may include iteratively identifying respective values for hyperparameters using analysis of the respective models performed using an acquisition function implementing constrained expected improvement on the respective models, training the machine learning system using the identified values to determine measures of accuracy and bias, and updating the respective models using the determined measures.

Abstract: Views may be generated for bias metrics or feature attribution captured in machine learning pipelines. A request to create a view of bias metrics or feature attribution may be received. The bias metrics or feature attribution may have been determined in a machine learning pipeline as part of executing a training job that specified the bias metrics or the feature attribution. A development application may access a data store that stores the bias metrics or the feature attribution determined in the machine learning pipeline. A view based on the bias metrics or feature attribution may be generated and provided.

Abstract: Bias metrics may be captured at different stages for training a machine learning model. A training job may specify bias metrics to capture at multiple different stages of a machine learning pipeline for a feature of a training data set used to train a machine learning model. The training job may be executed and the bias metrics determined at the stages as specified in the training job. The bias metrics for the different stages may be stored.

Abstract: Bias metrics and feature attribution may be monitored for a machine learning model. A request to enable monitoring for bias metrics or feature attribution may be received. Monitoring may be enabled to evaluate respective performance of inferences of a machine learning model according to the enabled bias metrics or feature attribution. If a divergence from reference data is detected, then a notification indicating the divergence may be sent.

Abstract: Feature attribution may be captured as part of a machine learning pipeline. A training job may include a request to determine feature attribution as part of a machine learning pipeline that trains a machine learning model from a training data set. A reference data set for determining the feature attribution of the machine learning model may be identified. The feature attribution may be determined based on the reference data set. The feature attribution of the trained machine learning model may be stored.