Abstract: Various methods and processes, apparatuses or systems, and media for performing a conformal prediction on an initial set of finite options generated by a large language model (LLM) are disclosed. The present disclosure provides receiving, via the LLM, a problem input in natural language including a question or prompt to select among a finite set of answer options as well as the answer options themselves; generating, via the LLM, a plurality of logits, wherein each of the plurality of logits correspond to the first set of finite answer options; generating, by executing a conformal prediction optimization (CP-OPT) framework, a score function for the LLM; and generating, via the score function, a conformal prediction set based on the first set of finite answer options, in which the conformal prediction set has a second set of finite answer options that is less than the first set of finite answer options, for improving accuracy of the LLM in identifying a correct answer to the problem input.

Abstract: Various methods and processes, apparatuses/systems, and media for auditing and enforcing conditional fairness in a machine learning model (MLM) are disclosed. A processor identifies features associated with the model; determines a first joint distribution of model outputs and a feature based on a first level of a particular one of the features and a second joint distribution of model outputs and a feature based on a second level of the particular feature; calculates corresponding conditional demographic disparity (CDD) between the first joint distribution and the second joint distribution at both levels; aggregates each CDD into a single CDD value; audits the conditional fairness by computing a distance between the first joint distribution and the second joint distribution based on the single CDD value; computes, based on the distance, a regularizer that reduces the CDD; and enforces the conditional fairness by applying the regularizer to the MLM.

Abstract: A method and a system for pre-processing data for algorithmic fairness via optimal transport in order to reduce disparities in classification datasets without modifying the original data are provided. The method includes: receiving a first dataset that includes a set of samples, each respective sample including a first coordinate that relates to sensitive demographic features, a second coordinate that relates to decision-making features, and a third coordinate that relates to a decision outcome that is generated by a machine learning model; determining a demographic parity constraint to be applied to the first dataset; computing a set of respective sample-level weights that correspond to each sample; and generating a second dataset by applying the set of respective sample-level weights to the first dataset The computation of the sample-level weights includes minimizing a Wasserstein distance between the first dataset and a weighted version thereof while satisfying the demographic parity constraint.

Abstract: A method and system for accounting for missing not-at-random (MNAR) data in training dataset via Bayesian regularization are disclosed. The method includes acquiring historical data of an organization, the historical data including the MNAR data. The method further includes performing estimation for two of at least three unknown quantities based on the historical data, and injecting quantitative information for remaining one of the at least three unknown quantities based on qualitative information regarding nature of missingness. Lastly, the method reassembles the estimation for two of the at least three unknown quantities and injected quantitative information for the remaining one of the at least three unknown quantities, to provide a modified function.

Abstract: A method and a system for obtaining conditional demographic parity in the construction of a data-driven model are provided. The method includes: identifying features associated with the model; determining a first joint distribution of model outputs and a feature based on a first level of a particular one of the features and a second joint distribution of model outputs and a feature based on a second level of the particular feature; computing a bi-causal transport distance between the first joint distribution and the second joint distribution; computing a regularizer based on the bi-causal transport distance; and applying the regularizer to the model.