Abstract: A system, electronic device and method for improved neural network training are provided. The electronic device includes: a processor, a memory storing a Generative adversarial network (GAN) to learn from unlabeled data by engaging a generative model in an adversarial game with a discriminator; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for training the Generative adversarial network using a regularizer to encourage the discriminator to properly use its capacity and hidden representations of the discriminator to have high entropy.

Abstract: Variational Autoencoders (VAEs) have been shown to be effective in modeling complex data distributions. Conventional VAEs operate with fully-observed data during training. However, learning a VAE model from partially-observed data is still a problem. A modified VAE framework is proposed that can learn from partially-observed data conditioned on the fully-observed mask. A model described in various embodiments is capable of learning a proper proposal distribution based on the missing data. The framework is evaluated for both high-dimensional multimodal data and low dimensional tabular data.

Abstract: Methods, systems, and techniques for density ratio estimation of data that includes a covariate variable (W) and a treatment variable (T). The density ratio estimation may be performed using a transformer-based architecture, and the density ratio may be used to control confounding bias in the data. An electronic representation of the data is obtained. At first and second self-attention layers, respectively, covariate variable embeddings based on the data representing the covariate variable and treatment variable embeddings based on the data representing the treatment variable are determined. Cross-attention embeddings based on the covariate and treatment variable embeddings are then determined at a cross-attention layer. At a linear layer and based on the cross-attention embeddings, a density ratio is estimated. The self-attention layers, cross-attention layer, and linear layer are trained using a loss function that determines a loss between an output of the linear layer and the density ratio.

Abstract: A computer implemented method is described for conducting text sequence machine learning, the method comprising: receiving an input sequence x=[x1, x2, . . . , xn], to produce a feature vector for a series of hidden states hx=[h1, h2, . . . , hn], wherein the feature vector for the series of hidden states hx is generated by performing pooling over a temporal dimension of all hidden states output by the encoder machine learning data architecture; and extracting from the series of hidden states hx, a mean and a variance parameter, and to encapsulate the mean and the variance parameter as an approximate posterior data structure.

Abstract: A system is provided comprising a processor and a memory storing instructions which configure the processor to process an original sentence structure through an encoder neural network to decompose the original sentence structure into an original semantics component and an original syntax component, process the original syntax component through a syntax variation autoencoder (VAE) to receive a syntax mean vector and a syntax covariance matrix, obtain a sampled syntax value from a syntax Gaussian posterior parameterized by the syntax mean vector and the syntax covariance matrix, process the original semantics component through a semantics VAE to receive a semantics mean vector and a semantics covariance matrix, obtain a sampled semantics vector from the Gaussian semantics posterior parameterized by the semantics mean vector and the semantics covariance matrix, and process the sampled syntax vector and the sampled semantics vector through a decoder neural network to compose a new sentence.

Abstract: A system, electronic device and method for improved neural network training are provided. The improved system is adapted for tracking long range dependence in sequential data during training, and includes bootstrapping a lower bound on the mutual information (MI) over groups of variables (segments or sentences) and subsequently applying the bound to encourage high MI.

Abstract: An improved computer implemented method and corresponding systems and computer readable media for improving performance of a deep neural network are provided to mitigate effects related to catastrophic forgetting in neural network learning. In an embodiment, the method includes storing, in memory, logits of a set of samples from a previous set of tasks (D1); and maintaining classification information from the previous set of tasks by utilizing the logits for matching during training on a new set of tasks (D2).

Abstract: A variational hyper recurrent neural network (VHRNN) can be trained by, for each step in sequential training data: determining a prior probability distribution for a latent variable from a prior network of the VHRNN using an initial hidden state; determining a hidden state from a recurrent neural network (RNN) of the VHRNN using an observation state, the latent variable and the initial hidden state; determining an approximate posterior probability distribution for the latent variable from an encoder network of the VHRNN using the observation state and the initial hidden state; determining a generating probability distribution for the observation state from a decoder network of the VHRNN using the latent variable and the initial hidden state; and maximizing a variational lower bound of a marginal log-likelihood of the training data. The trained VHRNN can be used to generate sequential data.

Abstract: Systems, methods, and computer readable media are described to train a compressed neural network with high robustness. The neural network is first adversarially pre-trained with both original data as well as data perturbed by adversarial attacks for some epochs, then “unimportant” weights or filters are pruned through criteria based on their magnitudes or other method (e.g., Taylor approximation of the loss function), and the pruned neural network is retrained with both clean and perturbed data for more epochs.

Abstract: A method for acquiring measurements for a data structure corresponding to an array of variable includes: selecting a subset of elements from the data structure; measuring a sampled value for each of the selected subset of elements; storing each of the sampled values in a K-nearest neighbour (KNN) database and labelling the sampled value as certain; generating a predicted value data structure where each predicted element is generated as the value of its nearest neighbor based on the values stored in the KNN database; for each predicted element: retrieve the predicted element's X nearest neighbours for the sampled value in the KNN database, and when a value of the X nearest neighbours is the same as the predicted element, the predicted element is labelled as certain, otherwise the predicted element is labelled the values as uncertain; and repeating until all elements are labelled as certain.

Abstract: A computer system and method for answering a natural language question is provided. The system comprises at least one processor and a memory storing instructions which when executed by the processor configure the processor to perform the method. The method comprises receiving a natural language question, generating a SQL query based on the natural language question, generating an explanation regarding a solution to the natural language question as answered by the SQL query, and presenting the solution and the explanation.

Abstract: Systems and methods of automatically generating a coherence score for text data is provided. The approach includes receiving a plurality of string tokens representing decomposed portions of the target text data object. A trained neural network is provided that has been trained against a plurality of corpuses of training text across a plurality of topics. The string tokens are arranged to extract string tokens representing adjacent sentence pairs of the target text data object. For each adjacent sentence pair, the neural network generates a local coherence score representing a coherence level of the adjacent sentence pair of the target text data object, which are then aggregated for each adjacent sentence pair of the target text data object to generate a global coherence score for the target text data object.

Abstract: A machine learning failure discriminator machine is described, along with corresponding systems, methods, and non-transitory computer readable media. The approach operates in relation to an iterative machine learning model and includes a phased approach to extract p-values from the iterative machine learning model based on modified versions of the training or validation data sets. The p-values are then used to identify whether various null hypotheses can be rejected, and accordingly, to generate an output data structure indicative of an estimated failure reason, if any. The output data structure may be made available on an API or on a graphical user interface.

Abstract: A platform, device and process is provided for visual construction of operations for data querying. In particular, embodiments described herein provides a platform, device and process for visual construction of nested operations for data querying. The visual construction is a display of one or more projected data spaces enabling a selection of data indicators on the display. The selection is conducted graphically on the visual construction and the system is configured to translate the selection to generate and conduct a query operating visually on the visualized (e.g., projected) data space. The visual data space includes distinct views of the plurality of multi-dimensionality data points mapped to reduced-dimensionality data points with a transformation function associated with each view. The selections are used to augment the multi-dimensionality data points with one or more additional dimensions to track the selections and to perform operations and visualizations.

Abstract: An electronic device for neural network training includes at least one processor and one or more memories configured to provide or train: a generative adversarial network (GAN) using a generator and a discriminator for: receiving a plurality of training cases; and training the generative adversarial network, based on the plurality of training cases, to classify the training cases; wherein the generator generates hard negative examples for the discriminator.

Abstract: Variational Autoencoders (VAEs) have been shown to be effective in modeling complex data distributions. Conventional VAEs operate with fully-observed data during training. However, learning a VAE model from partially-observed data is still a problem. A modified VAE framework is proposed that can learn from partially-observed data conditioned on the fully-observed mask. A model described in various embodiments is capable of learning a proper proposal distribution based on the missing data. The framework is evaluated for both high-dimensional multimodal data and low dimensional tabular data.

Abstract: A computer system and method for machine text generation is provided. The system comprises at least one processor and a memory storing instructions which when executed by the processor configure the processor to perform the method.

Abstract: A computer implemented method is described for conducting text sequence machine learning, the method comprising: receiving an input sequence x=[x1, x2, . . . , xn], to produce a feature vector for a series of hidden states hx=[h1, h2, . . . , hn], wherein the feature vector for the series of hidden states hx is generated by performing pooling over a temporal dimension of all hidden states output by the encoder machine learning data architecture; and extracting from the series of hidden states hx, a mean and a variance parameter, and to encapsulate the mean and the variance parameter as an approximate posterior data structure.

Abstract: A variational hyper recurrent neural network (VHRNN) can be trained by, for each step in sequential training data: determining a prior probability distribution for a latent variable from a prior network of the VHRNN using an initial hidden state; determining a hidden state from a recurrent neural network (RNN) of the VHRNN using an observation state, the latent variable and the initial hidden state; determining an approximate posterior probability distribution for the latent variable from an encoder network of the VHRNN using the observation state and the initial hidden state; determining a generating probability distribution for the observation state from a decoder network of the VHRNN using the latent variable and the initial hidden state; and maximizing a variational lower bound of a marginal log-likelihood of the training data. The trained VHRNN can be used to generate sequential data.

Abstract: There is provided a neural network system for detection of domain generation algorithm generated domain names, the neural network system comprising: an input receiver configured for receiving domain names from one or more input sources; a convolutional neural network unit including one or more convolutional layers, the convolutional unit configured for receiving the input text and processing the input text through the one or more convolutional layers; a recurrent neural network unit including one or more long short term memory layers, the recurrent neural network unit configured to process the output from the convolutional neural network unit to perform pattern recognition; and a classification unit including one or more classification layers, the classification unit configured to receive output data from the recurrent neural network unit to perform a determination of whether the input text or portions of the input text are DGA-generated or benign domain names.