Abstract: In some implementations, a computing machine accesses an artificial intelligence model and a dataset for the artificial intelligence model, the dataset comprising at least one datapoint. The computing machine identifies a feature group used by the artificial intelligence model, the feature group comprising at least two features having a similarity with one another exceeding a similarity threshold, wherein the feature group comprises a subset of the features used by the artificial intelligence model. The computing machine determines an overall influence value for the feature group on an output of the artificial intelligence model applied to the dataset. The computing machine provides an output representing the overall influence value.

Abstract: A computer accesses an artificial intelligence (AI) model, a labeled in-sample (IS) dataset, and an unlabeled out-of-sample (OOS) dataset, the labeled IS dataset storing IS input values and corresponding IS output values, the unlabeled OOS dataset storing OOS input values but not corresponding OOS output values. The computer modifies, via importance sampling and based on a likelihood that a given datapoint from the IS dataset is associated with the OOS dataset, weights of multiple datapoints in the labeled IS dataset to generate a weighted IS dataset. The computer calculates an estimated performance metric of the AI model on the OOS dataset using at least a subset of datapoints in the weighted IS dataset. The computer provides an output representing the estimated performance metric of the AI model on the OOS dataset.

Abstract: Computer accesses training dataset with plurality of datapoints, each datapoint having input vector of feature values and output value. Training dataset is for training machine learning engine to predict the output value based on the input vector of feature values. The computer stores the training dataset as a two-dimensional vector with rows representing datapoints and columns representing features. The computer computes, for each feature value, a QII (quantitative input influence) value measuring a degree of influence that the feature exerts on the output value. For each datapoint from at least a subset of the plurality of datapoints, the computer (i) determines whether the QII value for each feature value in the input vector is within a predefined range, and (ii) upon determining that the QII value for a given feature value in the input vector is not within the predefined range: adjusts the training dataset or the machine learning engine.

Abstract: A computing machine receives a representation of a machine learning model, a representation of a first data segment, and a representation of a second data segment. The computing machine computes an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment. The computing machine determines a set of reasons for the computed output difference based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference. The computing machine provides an output representing the set of reasons.

Abstract: A computing machine accesses a set of intermediate artificial neurons in a deep neural network. The deep neural network is fully or partially trained. The computing machine computes, for each artificial neuron in the set of intermediate artificial neurons, an influence score based on an average gradient of an output quantity of interest with respect to the artificial neuron across a plurality of inputs weighted by a probability of each input. The computing machine provides an output associated with the computed influence scores.

Abstract: The subject disclosure relates to devices, systems, and methods for algorithmic transparency into algorithmic decision-making systems. In non-limiting aspects, the disclosed subject matter facilitates generating a set of intervention inputs for an algorithmic decision-making system, observing the outcomes of the algorithmic decision-making system, and determining Quantitative Input Influence (QII) measures for the algorithmic decision-making system, wherein the at least one QII measure describes degree of influence of inputs on outcomes of the algorithmic decision-making system. In further non-limiting aspects, the disclosed subject matter facilitates generating transparency reports related to the QII measures, including transparency reports, regarding inputs, regarding individuals, and regarding groups of individuals, while maintaining privacy. Further non-limiting embodiments are provided that illustrate the advantages and flexibility of the disclosed subject matter.

Abstract: An apparatus and method for establishing a trusted path between a user interface and a trusted executable, wherein the trusted path includes a hypervisor and a driver shim. The method includes measuring an identity of the hypervisor; comparing the measurement of the identity of the hypervisor with a policy for the hypervisor; measuring an identity of the driver shim; comparing the measurement of the identity of the driver shim with a policy for the driver shim; measuring an identity of the user interface; comparing the measurement of the identity of the user interface with a policy for the user interface; and providing a human-perceptible indication of whether the identity of the hypervisor, the identity of the driver shim, and the identity of the user interface correspond with the policy for the hypervisor, the policy for the driver shim, and the policy for the user interface, respectively.

Abstract: A computer including a processor and a verification device. The processor in the computer performs the steps of authenticating a secure connection between a hypervisor and the verification device, measuring the identity of at least a portion of a select guest before the select guest executes any instruction, and sending a measurement of the identity of the select guest to the verification device. The verification device compares the policy stored in the verification device with the measurement of the select guest received by the verification device. The steps of authenticating, measuring, sending, and comparing are performed after receiving a signal indicative of a request to execute the select guest and without rebooting the computer.

Abstract: An apparatus and method for establishing a trusted path between a user interface and a trusted executable, wherein the trusted path includes a hypervisor and a driver shim. The method includes measuring an identity of the hypervisor; comparing the measurement of the identity of the hypervisor with a policy for the hypervisor; measuring an identity of the driver shim; comparing the measurement of the identity of the driver shim with a policy for the driver shim; measuring an identity of the user interface; comparing the measurement of the identity of the user interface with a policy for the user interface; and providing a human-perceptible indication of whether the identity of the hypervisor, the identity of the driver shim, and the identity of the user interface correspond with the policy for the hypervisor, the policy for the driver shim, and the policy for the user interface, respectively.

Abstract: A new and useful updating tool is provided for indexing electronic text in a highly integrated electronic text indexing and search system. The integrated system encompasses a concept model, a query model, an enhanced markup tool, a user interface, and a search engine. A domain expert utilizes the integrated system to efficiently and effectively indexing electronic text for precise and fast retrieval by a search engine. The updating tool aids the indexing process as a consequence of changes in the query model, to an indexed document, or both. The updating tool comprises a document change tool and a query model change tool as well as an enhanced markup tool and an enhanced query model tool. The updating tool guides the domain expert by suggesting which indices need to be updated and flags index entries that need to be removed.

Abstract: A new and useful updating tool is provided for indexing electronic text in a highly integrated electronic text indexing and search system. The integrated system encompasses a concept model, a query model, an enhanced markup tool, a user interface, and a search engine. A domain expert utilizes the integrated system to efficiently and effectively indexing electronic text for precise and fast retrieval by a search engine. The updating tool aids the indexing process as a consequence of changes in the query model, to an indexed document, or both. The updating tool comprises a document change tool and a query model change tool as well as an enhanced markup tool and an enhanced query model tool. The updating tool guides the domain expert by suggesting which indices need to be updated and flags index entries that need to be removed.