Abstract: Methods and computer systems improve a trained base deep neural network by structurally changing the base deep neural network to create an updated deep neural network, such that the updated deep neural network has no degradation in performance relative to the base deep neural network on the training data. The updated deep neural network is subsequently training. Also, an asynchronous agent for use in a machine learning system comprises a second machine learning system ML2 that is to be trained to perform some machine learning task. The asynchronous agent further comprises a learning coach LC and an optional data selector machine learning system DS. The purpose of the data selection machine learning system DS is to make the second stage machine learning system ML2 more efficient in its learning (by selecting a set of training data that is smaller but sufficient) and/or more effective (by selecting a set of training data that is focused on an important task).

Abstract: A machine learning system includes a coach machine learning system that uses machine learning to help a student machine learning system learn its system. By monitoring the student learning system, the coach machine learning system can learn (through machine learning techniques) “hyperparameters” for the student learning system that control the machine learning process for the student learning system. The machine learning coach could also determine structural modifications for the student learning system architecture. The learning coach can also control data flow to the student learning system.

Abstract: A multi-stage machine learning and recognition system comprises multiple individual machine learning systems arranged in multiple stages, where data is passed from a machine learning system in one stage to one or more machine learning systems in a subsequent, higher-level stage of the structure according to the logic of the machine learning system. The multi-stage machine learning system can be arranged in a final stage and one or more non-final stages, where the one or more non-final stages direct data generally towards a selected one or more machine learning systems within the final stage, but less than all of the machine learning systems in the final stage. The multi-stage machine learning system can additionally include a learning coach and data management system, which is configured to control the distribution of data throughout the multi-stage structure of machine learning systems by observing the internal state of the structure.

Abstract: Computer systems and computer-implemented methods train and/or operate, once trained, a machine-learning system that comprises a plurality of generator-detector pairs. The machine-learning computer system comprises a set of processor cores and computer memory that stores software. When executed by the set of processor cores, the software causes the set of processor cores to implement a plurality of generator-detector pairs, in which: (i) each generator-detector pair comprises a machine-learning data generator and a machine-learning data detector; and (ii) each generator-detector pair is for a corresponding cluster of data examples respectively, such that, for each generator-detector pair, the generator is for generating data examples in the corresponding cluster and the detector is for detecting whether data examples are within the corresponding cluster.

Abstract: A machine learning system includes a coach machine learning system that uses machine learning to help a student machine learning system learn its system. By monitoring the student learning system, the coach machine learning system can learn (through machine learning techniques) “hyperparameters” for the student learning system that control the machine learning process for the student learning system. The machine learning coach could also determine structural modifications for the student learning system architecture. The learning coach can also control data flow to the student learning system.

Abstract: Computer-implemented systems and methods soft-tie learned parameters of a neural network(s). The soft-tying comprises: applying a common label to the first and second learned parameters; and as part of the training, and in response to the first and second learned parameters having the common label, applying a regularization penalty to a loss function for the first learned parameter upon a determination that the first learned parameter is different than the second learned parameter. The learned parameters can be connection weights, node biases, and/or parametric model statistics. The application of the regularization penalty can be influenced by a soft-tying hyperparameter.

Abstract: Methods and computer systems improve a trained base deep neural network by structurally changing the base deep neural network to create an updated deep neural network, such that the updated deep neural network has no degradation in performance relative to the base deep neural network on the training data. The updated deep neural network is subsequently training. Also, an asynchronous agent for use in a machine learning system comprises a second machine learning system ML2 that is to be trained to perform some machine learning task. The asynchronous agent further comprises a learning coach LC and an optional data selector machine learning system DS. The purpose of the data selection machine learning system DS is to make the second stage machine learning system ML2 more efficient in its learning (by selecting a set of training data that is smaller but sufficient) and/or more effective (by selecting a set of training data that is focused on an important task).

Abstract: Computer-implemented, machine-learning systems and methods relate to a combination of neural networks. The systems and methods train the respective member networks both (i) to be diverse and yet (ii) according to a common, overall objective. Each member network is trained or retrained jointly with all the other member networks, including member networks that may not have been present in the ensemble when a member is first trained.

Abstract: Computer systems and computer-implemented methods train a neural network, by: (a) computing for each datum in a set of training data, activation values for nodes in the neural network and estimates of partial derivatives of an objective function for the neural network for the nodes in the neural network; (b) selecting a target node of the neural network and/or a target datum in the set of training data; (c) selecting a target-specific improvement model for the neural network, wherein the target-specific improvement model, when added to the neural network, improves performance of the neural network for the target node and/or the target datum, as the case may be; (d) training the target-specific improvement model; (e) merging the target-specific improvement model with the neural network to form an expanded neural network; and (f) training the expanded neural network.

Abstract: A computer-implemented method of training an ensemble machine learning system comprising a plurality of ensemble members. The method includes selecting a shared objective and an objective for each of the ensemble members. The method further includes training each of the ensemble members according to each objective on a training data set, connecting an output of each of the ensemble members to a joint optimization machine learning system to form a consolidated machine learning system, and training the consolidated machine learning system according to the shared objective and the objective for each of the ensemble members on the training data set. The ensemble members can be the same or different types of machine learning systems. Further, the joint optimization machine learning system can be the same or a different type of machine learning system than the ensemble members.

Abstract: A machine learning system includes a coach machine learning system that uses machine learning to help a student machine learning system learn its system. By monitoring the student learning system, the coach machine learning system can learn (through machine learning techniques) “hyperparameters” for the student learning system that control the machine learning process for the student learning system. The machine learning coach could also determine structural modifications for the student learning system architecture. The learning coach can also control data flow to the student learning system.

Abstract: Computer-implemented systems and methods build ensembles for deep learning through parallel data splitting by creating and training an ensemble of up to 2n ensemble members based on a single base network and a selection of n network elements. The ensemble members are created by the “blasting” process, in which training data are selected for each of the up to 2n ensemble members such that each of the ensemble members trains with updates in a different direction from each of the other ensemble members. The ensemble members may also be trained with joint optimization.

Abstract: Systems and methods for analyzing a first machine learning system via a second machine learning system. The first machine learning system comprising a first objective function. The method includes connecting the first machine learning system to an input of the second machine learning system, which includes a second objective function for analyzing an internal characteristic of the first machine learning system. The method further includes providing a data item to the first machine learning system, collecting internal characteristic data from the first machine learning system associated with the internal characteristic, computing partial derivatives of the first objective function through the first machine learning system with respect to the data item, and computing partial derivatives of the second objective function through both the second machine learning system and the first machine learning system with respect to the collected internal characteristic data.

Abstract: A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

Abstract: A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

Abstract: Computer-implemented systems and methods build ensembles for deep learning through parallel data splitting by creating and training an ensemble of up to 2n ensemble members based on a single base network and a selection of n network elements. The ensemble members are created by the “blasting” process, in which training data are selected for each of the up to 2n ensemble members such that each of the ensemble members trains with updates in a different direction from each of the other ensemble members. The ensemble members may also be trained with joint optimization.

Abstract: A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

Abstract: A computer system uses a pool of predefined functions and pre-trained networks to accelerate the process of building a large neural network or building a combination of (i) an ensemble of other machine learning systems with (ii) a deep neural network. Copies of a predefined function node or network may be placed in multiple locations in a network being built. In building a neural network using a pool of predefined networks, the computer system only needs to decide the relative location of each copy of a predefined network or function. The location may be determined by (i) the connections to a predefined network from source nodes and (ii) the connections from a predefined network to nodes in an upper network. The computer system may perform an iterative process of selecting trial locations for connecting arcs and evaluating the connections to choose the best ones.

Abstract: Machine-learning data generators use an additional objective to avoid generating data that is too similar to any previously known data example. This prevents plagiarism or simple copying of existing data examples, enhancing the ability of a generator to usefully generate novel data. A formulation of generative adversarial network (GAN) learning as the mixed strategy minimax solution of a zero-sum game solves the convergence and stability problem of GANs learning, without suffering mode collapse.

Abstract: Computer systems and computer-implemented methods train a machine-learning regression system. The method comprises the step of generating, with a machine-learning generator, output patterns; distorting the output patterns of the generator by a scale factor to generate distorted output patterns; and training the machine-learning regression system to predict the scaling factor, where the regression system receives the distorted output patterns as input and learns and the scaling factor is a target value for the regression system. The method may further comprise, after training the machine-learning regression system, training a second machine-learning generator by back propagating partial derivatives of an error cost function from the regression system to the second machine-learning generator and training the second machine-learning generator using stochastic gradient descent.