Abstract: A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

Abstract: Computer-implemented systems and method train a generator and a discriminator, through machine learning, where the generator and discriminator are trained in an adversarial relationship using a simulated, multi-player game. The model parameters for the generator and the discriminator can be updated non-simultaneously. Also, the simulated, multi-player game may comprise a two-person, zero-sum game.

Abstract: Data-dependent node-to-node knowledge sharing to increase the interpretability of the activation pattern of one or more nodes in a neural network, is implemented by a set of knowledge sharing links. Each link may comprise a knowledge providing node or other source P and a knowledge receiving node R. A knowledge sharing link can impose a node-specific regularization on the knowledge receiving node R to help guide the knowledge receiving node R to have an activation pattern that is more easily interpreted. The specification and training of the knowledge sharing links may be controlled by a cooperative human-AI learning supervisor system in which a human and an artificial intelligence system work cooperatively to improve the interpretability and performance of the client system.

Abstract: Computer systems and computer-implemented methods train a neural network, by: (a) computing for each datum in a set of training data, activation values for nodes in the neural network and estimates of partial derivatives of an objective function for the neural network for the nodes in the neural network; (b) selecting a target node of the neural network and/or a target datum in the set of training data; (c) selecting a target-specific improvement model for the neural network, wherein the target-specific improvement model, when added to the neural network, improves performance of the neural network for the target node and/or the target datum, as the case may be; (d) training the target-specific improvement model; (e) merging the target-specific improvement model with the neural network to form an expanded neural network; and (f) training the expanded neural network.

Abstract: Machine-learning computer system breaks a neural network into a plurality of modules and tracks the training process module-by-module and datum-by-datum, recording auxiliary information during one iteration of the training process for retrieval during a later iteration. Based on this auxiliary information, the computer system can make decisions that can greatly reduce the amount of computation required by the training process. The auxiliary information allows the computer system to diagnose and fix problems that occur during the training process on a module-by-module and/or datum-by-datum basis.

Abstract: A diverse set of neural networks are trained to be individually robust against adversarial attacks and diverse in a manner that decreases the ability of an adversarial example to fool the full diverse set. The systems/methods use a diversity criterion that is specialized for measuring diversity in response to adversarial attacks rather than diversity in the classification results. Also, one or more networks can be trained that are less robust to adversarial attacks to use as a diagnostic to detect the presence of an adversarial attack. Also, node-to-node relation regularization links can be used to train diverse networks that are randomly selected from a family of diverse networks with exponentially many members.

Abstract: A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

Abstract: Computer-implemented systems and methods improve training of a neural network. Whether a target node is not decisive on a training data item is determined. Upon a determination that the target node is not decisive, a partial derivative of an objective for the target node is multiplied by a factor greater than 1.0 for the training data item. Determining whether the target node is not decisive can comprise determining whether a direction of the derivative is in a direction that would cause an update of learned parameters for the network to increase the difference between the activation value of the first target node for the training data item and a neutral activation value for the target node.

Abstract: Data-dependent node-to-node knowledge sharing to increase the interpretability of the activation pattern of one or more nodes in a neural network, is implemented by a set of knowledge sharing links. Each link may comprise a knowledge providing node or other source P and a knowledge receiving node R. A knowledge sharing link can impose a node-specific regularization on the knowledge receiving node R to help guide the knowledge receiving node R to have an activation pattern that is more easily interpreted. The specification and training of the knowledge sharing links may be controlled by a cooperative human-AI learning supervisor system in which a human and an artificial intelligence system work cooperatively to improve the interpretability and performance of the client system.

Abstract: Computer-implemented systems and method train a generator and a discriminator, through machine learning, where the generator and discriminator are trained in an adversarial relationship using a simulated, multi-player game. The model parameters for the generator and the discriminator can be updated non-simultaneously. Also, the simulated, multi-player game may comprise a two-person, zero-sum game.

Abstract: Computer systems and computer-implemented methods train a neural network, by: (a) computing for each datum in a set of training data, activation values for nodes in the neural network and estimates of partial derivatives of an objective function for the neural network for the nodes in the neural network; (b) selecting a target node of the neural network and/or a target datum in the set of training data; (c) selecting a target-specific improvement model for the neural network, wherein the target-specific improvement model, when added to the neural network, improves performance of the neural network for the target node and/or the target datum, as the case may be; (d) training the target-specific improvement model; (e) merging the target-specific improvement model with the neural network to form an expanded neural network; and (f) training the expanded neural network.

Abstract: Computer-implemented systems and methods improve training of a neural network. Whether a target node is not decisive on a training data item is determined. Upon a determination that the target node is not decisive, a partial derivative of an objective for the target node is multiplied by a factor greater than 1.0 for the training data item. Determining whether the target node is not decisive can comprise determining whether a direction of the derivative is in a direction that would cause an update of learned parameters for the network to increase the difference between the activation value of the first target node for the training data item and a neutral activation value for the target node.

Abstract: A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

Abstract: Computer systems and methods train a deep neural network through machine learning. In response to detection of a training condition, computer system replaces a target node of the network with a split detector compound node, where, prior to replacement, the target node detected a pattern that activated the target node beyond a specified threshold. The split detector compound node comprises first and second nodes, such that: the first node is activated when significant evidence exists in favor of detection of the pattern in inputs to the first node; and the second node is activated when significant evidence exists against detection of the pattern in inputs to the second node, such that activations of the first and second nodes are computed independently. After replacing the target node with the split detector compound node, training of the network through machine learning is resumed.

Abstract: Computer-implemented systems and method train a generator and a discriminator, through machine learning, where the generator and discriminator are trained in an adversarial relationship using a simulated, multi-player game. The model parameters for the generator and the discriminator can be updated non-simultaneously. Also, the simulated, multi-player game may comprise a two-person, zero-sum game.

Abstract: Systems and methods improve performance of a classifier, which comprises a neural network and is trained through machine learning. First and second scores are computed, by the classifier, for each a multiple data examples from a generator. The first score is indicative of whether the data example belongs to a first data cluster and the second score is indicative of whether the data example belongs to a second data cluster. The generator is trained with an objective such that, for each data example generated by the generator, the first and second scores computed by the classifier are equal. Partial derivatives from the classifier are back-propagated for multiple data examples generated by the generator, to obtain a vector, for each data example, that is orthogonal to a decision surface for the classifier. A problem with the classifier is detected based on changes in directions of the vectors.

Abstract: Data-dependent node-to-node knowledge sharing to increase the interpretability of the activation pattern of one or more nodes in a neural network, is implemented by a set of knowledge sharing links. Each link may comprise a knowledge providing node or other source P and a knowledge receiving node R. A knowledge sharing link can impose a node-specific regularization on the knowledge receiving node R to help guide the knowledge receiving node R to have an activation pattern that is more easily interpreted. The specification and training of the knowledge sharing links may be controlled by a cooperative human-AI learning supervisor system in which a human and an artificial intelligence system work cooperatively to improve the interpretability and performance of the client system.

Abstract: Computer systems and computer-implemented methods train a neural network iteratively training, through machine learning. The iterative training comprises imposing a first is-not-equal-to regularization link between first and second nodes, where imposing the first is-not-equal-to regularization link between the two nodes comprises adding, during back-propagation of partial derivatives through the neural network for a datum in a training data set, a first regularization cost to a network error loss function for the first node that is inversely proportional to a difference between an activation value for the first node for the datum and an activation value for the second node for the datum.

Abstract: A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

Abstract: Data-dependent node-to-node knowledge sharing to increase the interpretability of the activation pattern of one or more nodes in a neural network, is implemented by a set of knowledge sharing links. Each link may comprise a knowledge providing node or other source P and a knowledge receiving node R. A knowledge sharing link can impose a node-specific regularization on the knowledge receiving node R to help guide the knowledge receiving node R to have an activation pattern that is more easily interpreted. The specification and training of the knowledge sharing links may be controlled by a cooperative human-AI learning supervisor system in which a human and an artificial intelligence system work cooperatively to improve the interpretability and performance of the client system.