Systems and methods for training predictive models that ignore missing features

- Unlearn.AI, Inc.

One embodiment includes a method for generating representations of inputs with missing values. The method includes steps for receiving an input includes a set of one or more values for several features, wherein the set of values for at least one of the several features includes values for each of several points in time, and for identifying a missingness pattern of the input, wherein the missingness pattern for the at least one feature indicates whether the set of values is missing a value for each of the several points in time. The method further includes steps for determining a set of one or more transformation weights based on the missingness pattern, and transforming the input based on the determined transformation weights.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
CROSS-REFERENCE TO RELATED APPLICATIONS

The current application claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63/486,429 entitled “Systems and Methods for Training Predictive Models that Ignore Missing Features” filed Feb. 22, 2023. The disclosure of U.S. Provisional Patent Application No. 63/486,429 is hereby incorporated by reference in its entirety for all purposes.

FIELD OF THE INVENTION

The present invention generally relates to training and using predictive models and, more specifically, training and using predictive models that ignore missing features.

BACKGROUND

Machine Learning (ML) models are used in many settings to predict responses from observed input features. A textbook example of an ML problem is to classify an object, given an image of the object. Should some of the pixels in the image be obscured, rendering their intensity unknown, the object classification problem becomes more difficult. Missing/incomplete features (obscured pixels in the case of an image) challenge ML models in many disciplines.

SUMMARY OF THE INVENTION

Systems and methods for training and utilizing predictive models that ignore missing features in accordance with embodiments of the invention are illustrated. One embodiment includes a method for generating representations of inputs with missing values. The method includes steps for at a single layer in a multi-layer model receiving an input includes a set of one or more values for several features and identifying a missingness pattern of the input, wherein the missingness pattern indicates whether the set of values is missing a value for each of the several features. The method further includes determining a set of one or more transformation weights based on the missingness pattern and transforming the input based on the determined transformation weights.

In a further embodiment, determining a set of transformation weights based on the missingness pattern includes utilizing a set of one or more attention weights.

In still another embodiment, the missingness pattern is a binary vector.

In a still further embodiment, transforming the input is further based on an offset that depends on the missingness pattern.

In yet another embodiment, determining the set of transformation weights includes scaling the set of transformation weights.

In a yet further embodiment, transforming the input includes linearly mixing the missingness pattern with a fixed linear transformation.

In another additional embodiment, the method further includes steps for updating a model based on the transformed input.

In a further additional embodiment, the method further includes steps for generating a predictive output based on the transformed input.

One embodiment includes a non-transitory machine readable medium containing processor instructions for generating representations of inputs with missing values. Execution of the instructions by a processor causes the processor to perform a process that includes at a single layer in a multi-layer model receiving an input includes a set of one or more values for several features and identifying a missingness pattern of the input. The missingness pattern indicates whether the set of values is missing a value for each of the several features. The process further includes determining a set of one or more transformation weights based on the missingness pattern and transforming the input based on the determined transformation weights.

One embodiment includes a method for generating representations of inputs with missing values. The method includes steps for receiving an input includes a set of one or more values for several features, wherein the set of values for at least one of the several features includes values for each of several points in time, and for identifying a missingness pattern of the input, wherein the missingness pattern for the at least one feature indicates whether the set of values is missing a value for each of the several points in time. The method further includes steps for determining a set of one or more transformation weights based on the missingness pattern, and transforming the input based on the determined transformation weights.

In another embodiment again, determining a set of transformation weights based on the missingness pattern includes utilizing a set of one or more attention weights.

In a further embodiment again, utilizing a set of attention weights includes determining the set of attention weights using a set of one or more neural networks.

In still yet another embodiment, the missingness pattern is a binary vector.

In a still yet further embodiment, transforming the input is further based on an offset that depends on the missingness pattern.

In still another additional embodiment, determining the set of transformation weights includes scaling the set of transformation weights.

In a still further additional embodiment, transforming the input includes linearly mixing the missingness pattern with a fixed linear transformation.

In still another embodiment again, the method further includes steps for updating a model based on the transformed input.

In a still further embodiment again, the method further includes steps for generating a predictive output based on the transformed input.

One embodiment includes a non-transitory machine readable medium containing processor instructions for generating representations of inputs with missing values. Execution of the instructions by a processor causes the processor to perform a process that comprises receiving an input includes a set of one or more values for several features, wherein the set of values for at least one of the several features includes values for each of several points in time, and identifying a missingness pattern of the input, wherein the missingness pattern for the at least one feature indicates whether the set of values is missing a value for each of the several points in time. The process further includes determining a set of one or more transformation weights based on the missingness pattern, and transforming the input based on the determined transformation weights.

Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.

FIG. 1 conceptually illustrates an example of a process for generating dense representations of inputs with missing values.

FIG. 2 illustrates an example diagram of a MissAttention layer design in accordance with an embodiment of the invention.

FIG. 3 illustrates an example diagram of a MissAttention layer design in accordance with an embodiment of the invention.

FIG. 4 illustrates an example of a MissAttention system that trains and/or utilizes models based on data with missing values in accordance with an embodiment of the invention.

FIG. 5 illustrates an example of a MissAttention element that executes instructions to perform processes that train and/or utilize models based on data with missing values in accordance with an embodiment of the invention.

FIG. 6 illustrates an example of a MissAttention application for generating dense representations of inputs with missing values in accordance with an embodiment of the invention.

 DETAILED DESCRIPTION

Turning now to the drawings, systems and methods in accordance with many embodiments of the invention can train and utilize models with incomplete data by converting data with missing features to dense feature-complete representations. Models that can utilize data with missing values for features (or MissAttention models) can be trained to learn not only from the values that are present, but from the missingness of the values for different features.

In many cases, the features that are missing for a given observation varies randomly and the distribution of missing features may be itself informative. In recommendation problems, where the objective is to make personalized recommendations for individual users, voluntarily supplied features like user ratings and surveys can be important features to use. Voluntarily supplied features are often highly incomplete. In clinical trials, the aim is to predict a patient's disease progression based on a variety of scheduled tests, but some sub-studies may opt to skip some tests and participants in the trial might miss scheduled visits. Consequently, clinical trial datasets are rife with missing features. Even in the realm of sensor data analysis, where data is more reliably complete, there are examples of missing data that arise due communication and sensor failures.

The common approach to handle data with missing values is to “impute” values so that the dataset can be made complete before a second, separately trained model is applied to the imputed dataset. In contrast to the commonly employed approach, the second approach is to create a single model which can handle incomplete data. Systems and methods in accordance with certain embodiments of the invention create a single model which can handle incomplete data, thus following this second approach.

In a variety of embodiments, models can be deterministic (not random) and can be formulated with a neural network layer that is used to convert data with missing features to dense feature-complete representations. As a neural network layer, it can be broadly compatible with a wide range of neural network approaches. Systems and methods in accordance with certain embodiments of the invention are deterministic and can be readily combined with other neural network methods, as they are encapsulated as a neural network layer.

Systems and methods in accordance with some embodiments of the invention provide several benefits over existing methods and processes. For example, by utilizing a missingness pattern, rather than simply the features which are present, processes in accordance with a number of embodiments of the invention can modify the bias term based on the missingness of different sets of features. Other methods may attempt to create a dense representation based solely on the observed features, but such processes are unable to derive information from the points which are missing. Processes in accordance with various embodiments of the invention also allow for dense representations to be generated from a light weight, nearly linear layer and can be readily applied to continuous and/or discrete spaces. Although many of the examples described herein describe implementations in a single layer of a neural network, one skilled in the art will recognize that similar systems and methods can be implemented in two or more layers of one or more neural networks in various configurations, without departing from this invention.

In many realistic problem settings, features that are missing are as important as those that are observed. For instance, in clinical trials, a visit may be skipped because the patient is too ill or a test may be skipped due to a different standard of care at that particular health care facility. In either case, which tests or visits are missed may be by itself predictive of certain outcomes. Thus, systems and methods in accordance with a variety of embodiments of the invention allow for the pattern of missing features to impact the output of the neural network layer as much as the values which are observed. Borrowing the concept of “attention” from the image analysis and natural language processing communities, where certain emphasis or attention is preferentially placed on words or image regions, processes in accordance with many embodiments of the invention apply an emphasis or attention that depends on which features are missing.

Data is often incomplete, with some event records having more data present than others. Many machine learning methods, however, expect dense and complete input data. Systems and methods in accordance with a number of embodiments of the invention utilize a neural network layer (or MissAttention layer) to transform incomplete data to a complete, dense representation in such a way that the MissAttention layer can be easily co-trained with other layers in a neural network. The approach is computationally efficient, similar in complexity to a linear layer, and can serve a wide range of application areas from tabular data to sensor data to time series data. This scaling is significant, since other approaches are substantially more expensive. For example, in a mixture of experts approach, as the dimensionality M of x grows, the number of cases grows as 2M, which is not tractable. 30 dimensions would require over a billion experts. In contrast, MissAttention scales quadratically in space and time with respect to input dimension.

An example of a process for generating dense representations of inputs with missing values is conceptually illustrated in FIG. 1. In certain embodiments, processes for generating dense representations can be performed in a single layer of a model. Process 100 receives (105) an input. Inputs in accordance with numerous embodiments of the invention can include various types of data, such as (but not limited to) medical data, time-series data, marketing data, etc. In several embodiments, inputs include values for each of several different features, where each particular input may be missing values for one or more of the features.

Process 100 identifies (110) a missingness pattern based on the input. Missingness patterns in accordance with some embodiments of the invention indicate whether values are missing for the different features. In numerous embodiments, missingness patterns are represented as binary vectors.

Process 100 determines (115) a set of transformation weights based on the missingness pattern. Transformation weights in accordance with several embodiments of the invention are a weighting vector that depends on the missingness pattern of the inputs. In many embodiments, transformation weights can be scaled based on the missingness pattern. In various embodiments, the rows and/or columns of a fixed linear transformation can be scaled. Determining the set of transformation weights in accordance with numerous embodiments of the invention can include linearly mixing a fixed linear transformation with the missingness pattern.

Process 100 transforms (120) the input based on the determined transformation weights. In certain embodiments, transforming inputs is further based on an offset that can be determined based on the missingness pattern.

In numerous embodiments, processes for generating dense outputs can be used to train and/or utilize a model (e.g., a predictive model) when the training (or input) data may be missing values for features in the data set. Processes in accordance with some embodiments of the invention can update a model (or learn) based on transformed inputs.

While specific processes for generating dense representations of inputs with missing values are described above, any of a variety of processes can be utilized to generate dense representations as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted.

Systems and methods in accordance with some embodiments of the invention can ingest data with missing features and produce from it a representation free of missing features. In some embodiments, dense representations can be produced through a MissAttention “layer.” MissAttention layers in accordance with several embodiments of the invention are compatible with neural network software frameworks like PyTorch, Tensorflow, MXNet and the like. In some embodiments, ingested data and/or dense representations can be include time series tabular data. In various embodiments, parameters of the MissAttention “layer” can be co-trained alongside other layers. In several embodiments, processes can optimize the initialization of weights for training MissAttention layers.

A. MissAttention Layer

An example diagram of a MissAttention layer design in accordance with an embodiment of the invention is illustrated in FIG. 2. In this example, data flows from left to right, beginning with input x. The trapezoidal shapes 205 and 210 depict linear layers. Attention 215 based on the missing mask m modifies the weights of the upper data flow path. In this example, a “simple impute” is performed on the input x Imputation in accordance with numerous embodiments of the invention can include various strategies, such as (but not limited to) zero filling, mean filling, back-filling, etc.

In a variety of embodiments, processes for producing a representation of the data that lacks missing features is encapsulated as neural network “layer”. Layers are modular components which can be reused in different neural networks. MissAttention layers in accordance with many embodiments of the invention can be built using linear transformations and/or element-wise non linearities, which are readily available in many neural network software frameworks. In various embodiments, the forward pass of a MissAttention layer produces an output vector y∈N from some input x∈M. In numerous embodiments, MissAttention layers can be given by a linear transformation of x (e.g., equation 3), where the weights of the transformation depend on the missingness pattern of x. Processes in accordance with some embodiments of the invention indicate missing elements of x through an indicator variable m, where mi=1 when xi is missing and mj=0 when xj is not missing. The transform is parameterized by C∈N×M, A∈M×M, and B∈N×M, and a bias term a∈N:

y = a + C ⁢ m + B ( ( x ∘ ( 1 - m ) ) ︸ x ~ ∘ φ ⁡ ( Am ) ) ( 1 ) = a + Cm + B ⁡ ( x ~ ∘ φ ⁡ ( Am ) ) ( 2 ) = a + Cm + B ⁢ diag ( φ ⁢ ( Am ) ) ⁢ x ~ ( 3 )
=a+Cm+B({tilde over (x)}∘φ(Am))  (2)
=a+Cm+Bdiag(φ(Am)){tilde over (x)}  (3)
where in the above, {tilde over (x)} is the zero-imputed version of x. Here ∘ indicates an element-wise product and diag(v) indicates a diagonal matrix with diagonal elements given by the vector argument v. φ is a potentially non-linear function that is applied to the elements of Am elementwise. Therefore φ(Am) is a weighting vector that depends on the missingness pattern of x. The function φ can include (but is not limited to) the exponential function and/or softmax. In a variety of embodiments, the term Cm adds an offset the purely depends on the missing pattern in x, and a is a pure bias term.

Stacking the input vectors x into a matrix X, the missing indicators as M, and the outputs as Y, the layer's operation can be written as,
Y=a+CM+B(φ(AM)∘{tilde over (X)}).  (4)
In the database-inspired language of attention, A is the matrix of keys, M the matrix of queries, and {tilde over (X)} the values.

In certain embodiments, missAttention layers can be formulated as follows:
y(1)=a+Cm+diag(φ1(A1m))Bdiag(φ2(A2m)){tilde over (x)}
y(2)=a+Cm+B(m,x){tilde over (x)}

The first extension, denoted y(1), adds an additional diagonal missingness-dependent term diag(φ1). The effect of this is to scale the rows of the otherwise fixed linear transformation B, while diag(φ2) exists to scale the columns. In the second case, y(2), we consider a matrix-valued function , such as the soft-max kernel or squared exponential kernel, which accepts the missingness vetor m and the input with missingness x as arguments. This generalization allows the missingness vector linearly mix B, instead of simply scaling its rows and/or columns.

B. Scalability of MissAttention

Systems and methods in accordance with several embodiments of the invention provide a computationally efficient approximation for estimating the response y given x. To illustrate the efficiency of MissAttention, note that other more straight-forward solutions to the problem of missing data fail to scale to higher dimensional problems. Consider a simple linear regression problem, with a zero-centered continuous response y that depends on two inputs a and b. Denote all observations as y and all the training inputs as x=[a,b]. With only two inputs, 4 linear models can be enumerated, each optimal for the different missing input possibilities. Using the same notation as above, denote a missing response m=[1,0] to indicate a is missing and m=[0,1] for when b is missing. The four possible missing states are {[1,0], [0,1], [1,1], [0,0]}. The best linear model for each case is:

y = { cov ⁡ ( y , b ) var ⁡ ( b ) ⁢ b given ⁢ m = [ 1 , 0 ] cov ⁡ ( y , a ) var ⁡ ( a ) ⁢ a given ⁢ m = [ 0 , 1 ] 0 given ⁢ m = [ 1 , 1 ] var ⁡ ( b ) ⁢ cov ⁡ ( y , a ) - cov ⁡ ( a , b ) ⁢ cov ⁡ ( b , y ) var ⁡ ( a ) ⁢ var ⁡ ( b ) + cov 2 ( a , b ) ⁢ a + var ⁡ ( a ) ⁢ cov ⁡ ( y , b ) - cov ⁡ ( a , b ) ⁢ cov ⁡ ( a , y ) var ⁡ ( a ) ⁢ var ⁡ ( b ) + cov 2 ( a , b ) ⁢ b given ⁢ m = [ 0 , 0 ]

Such a model is a mixture of experts, with each expert specializing in a different missingness pattern. As the dimensionality M of x grows, however, the number of cases grows as 2M, which is not tractable. 30 dimensions would require over a billion experts. In contrast, MissAttention scales quadratically in space and time with respect to input dimension.

C. Systems and Methods for Extension to Time Series Tabular Data

Time is a common index dimension encountered in tabular data. For instance, subjects arriving to a clinical trial appear at multiple times. Businesses that report earnings and various financials also report results on a cadence. A diagram of an example of Time Dependent MissAttention layer design is illustrated in FIG. 3. Data flows from left to right, beginning with input x(t). The trapezoidal shapes 305 and 310 in this example depict linear layers, where it can be seen that attention 315 modifies the representation of the data affected by the upper linear layer based on the missing mask m(t). “Simple Impute” in accordance with many embodiments of the invention can refer to various strategies, such as (but not limited to) zero filling (featured in equation 2), mean filling, back-filling, etc. Note that in the case of time dependent miss-attention, the attention weight ϕ may no longer be a simple function. In a number of embodiments, attention weights are time-dependent neural networks that depend on a missingness pattern as a function of time and the parameters of the network θ.

To handle such data, MissAttention can be generalized as follows:

y ⁡ ( t ) = a + C ⁢ m ⁡ ( t ) + B ( ( x ⁡ ( t ) ∘ ( 1 - m ⁡ ( t ) ) ) ︸ x ~ ( t ) ) ∘ φ ( m ⁡ ( t ) ,   θ ,   t ) )
where y(t), m(t), φ( . . . , t) are respectively the multivariate feature representation, missingness pattern, and/or attention weights which can depend on time. The time dependence may be itself encoded as a neural network or as a fixed parameterization if the time steps are known and fixed in advance.
D. Systems and Methods for Optimal Co-Training of MissAttention

A key consideration for neural network layers is how the parameters of the layer are initialized. In various embodiments, parameters of a MissAttention layer can be initialized to optimize training of the layer. For a centered observable y, the desired output may be zero when all elements of m are 1, because in this case, nothing is known and the mean should be reported. Given the simplified form y=a+Cm+B({tilde over (x)}∘φ(Am)), when m=1, each row of C should sum to −aj, since {tilde over (x)} will zero out the third term. If the bias term is initialized to zero, then each row of C should sum to zero. Notice that if all elements of A, within the φ term, are negative, then φ will be close to zero for all rows. This would push the response of all output rows towards zero, which would be a good initialization for a centered observable.

Initializing weights in accordance with a variety of embodiments of the invention can include:

Initializing C randomly such that all rows sum to zero;

Initializing a to zero;

Initializing A to random, strictly negative values, when non-linearity of a results in positive only values; and

Requiring φ to take value 1 when all values of m are zero.

While a specific process for initializing a MissAttention layer in accordance with several embodiments of the invention is described above, any of a variety of processes can be utilized to initialize layers as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted.

E. Systems for MissAttention

1. MissAttention System

An example of a MissAttention system that trains and/or utilizes models based on data with missing values in accordance with an embodiment of the invention is illustrated in FIG. 4. Network 400 includes a communications network 460. The communications network 460 is a network such as the Internet that allows devices connected to the network 460 to communicate with other connected devices. Server systems 410, 440, and 470 are connected to the network 460. Each of the server systems 410, 440, and 470 is a group of one or more servers communicatively connected to one another via internal networks that execute processes that provide cloud services to users over the network 460. One skilled in the art will recognize that a MissAttention system may exclude certain components and/or include other components that are omitted for brevity without departing from this invention.

For purposes of this discussion, cloud services are one or more applications that are executed by one or more server systems to provide data and/or executable applications to devices over a network. The server systems 410, 440, and 470 are shown each having three servers in the internal network. However, the server systems 410, 440 and 470 may include any number of servers and any additional number of server systems may be connected to the network 460 to provide cloud services. In accordance with various embodiments of this invention, a MissAttention system that uses systems and methods that train and/or utilize models based on data with missing values in accordance with an embodiment of the invention may be provided by a process being executed on a single server system and/or a group of server systems communicating over network 460.

Users may use personal devices 480 and 420 that connect to the network 460 to perform processes that train and/or utilize models based on data with missing values in accordance with various embodiments of the invention. In the shown embodiment, the personal devices 480 are shown as desktop computers that are connected via a conventional “wired” connection to the network 460. However, the personal device 480 may be a desktop computer, a laptop computer, a smart television, an entertainment gaming console, or any other device that connects to the network 460 via a “wired” connection. The mobile device 420 connects to network 460 using a wireless connection. A wireless connection is a connection that uses Radio Frequency (RF) signals, Infrared signals, or any other form of wireless signaling to connect to the network 460. In the example of this figure, the mobile device 420 is a mobile telephone. However, mobile device 420 may be a mobile phone, Personal Digital Assistant (PDA), a tablet, a smartphone, or any other type of device that connects to network 460 via wireless connection without departing from this invention.

As can readily be appreciated the specific computing system used to train and/or utilize models based on data with missing values is largely dependent upon the requirements of a given application and should not be considered as limited to any specific computing system(s) implementation.

2. MissAttention Element

An example of a MissAttention element that executes instructions to perform processes that train and/or utilize models based on data with missing values in accordance with an embodiment of the invention is illustrated in FIG. 5. MissAttention elements in accordance with many embodiments of the invention can include (but are not limited to) one or more of mobile devices, cameras, and/or computers. MissAttention element 500 includes processor 505, peripherals 510, network interface 515, and memory 520. One skilled in the art will recognize that a MissAttention element may exclude certain components and/or include other components that are omitted for brevity without departing from this invention.

The processor 505 can include (but is not limited to) a processor, microprocessor, controller, or a combination of processors, microprocessor, and/or controllers that performs instructions stored in the memory 520 to manipulate data stored in the memory. Processor instructions can configure the processor 505 to perform processes in accordance with certain embodiments of the invention. In various embodiments, processor instructions can be stored on a non-transitory machine readable medium.

Peripherals 510 can include any of a variety of components for capturing data, such as (but not limited to) cameras, displays, and/or sensors. In a variety of embodiments, peripherals can be used to gather inputs and/or provide outputs. MissAttention element 500 can utilize network interface 515 to transmit and receive data over a network based upon the instructions performed by processor 505. Peripherals and/or network interfaces in accordance with many embodiments of the invention can be used to gather inputs that can be used to train and/or utilize models based on data with missing values.

Memory 520 includes a MissAttention application 525, missing value data 530, and model data 535. MissAttention applications in accordance with several embodiments of the invention can be used to train and/or utilize models based on data with missing values. An example of a MissAttention application in accordance with several embodiments of the invention is described in greater detail below.

Missing value data in accordance with a variety of embodiments of the invention can include various types of data that may be missing values for certain features, such as (but not limited to) medical data, time series data, marketing data, recommendation data, etc.

In several embodiments, model data can store various parameters and/or weights for various models or MissAttention layers that can be used for various processes as described in this specification. Model data in accordance with many embodiments of the invention can be updated through training on missing value data captured on a MissAttention element or can be trained remotely and updated at a MissAttention element.

Although a specific example of a MissAttention element 500 is illustrated in this figure, any of a variety of MissAttention elements can be utilized to perform processes for generating dense representations of inputs with missing values similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.

3. MissAttention Application

An example of a MissAttention application for generating dense representations of inputs with missing values in accordance with an embodiment of the invention is illustrated in FIG. 6. MissAttention application 600 includes missingness engine 605, transformation engine 610, and output engine 615. One skilled in the art will recognize that a MissAttention application may exclude certain components and/or include other components that are omitted for brevity without departing from this invention. Missingness engines in accordance with various embodiments of the invention determine missingness as described throughout this specification. In many embodiments, transformation engines transform inputs to dense representations based on missingness, as described throughout this description. Output engines in accordance with several embodiments of the invention can provide a variety of outputs to a user, including (but not limited to) dense representations of inputs that are missing values. Dense representations in accordance with several embodiments of the invention can serve as inputs to additional layers of a model.

Although a specific example of a MissAttention application is illustrated in this figure, any of a variety of MissAttention applications can be utilized to perform processes for generating dense representations of inputs with missing values similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.

Although specific methods of generating dense representations of inputs with missing values are discussed above, many different methods of generating dense representations of inputs with missing values can be implemented in accordance with many different embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

1. A method for generating representations of inputs with missing values, the method comprising:

receiving an input comprising a set of one or more values for a plurality of features, wherein the set of values for at least one of the plurality of features comprises values for each of a plurality of points in time;
identifying a missingness pattern of the input, wherein the missingness pattern for the at least one feature indicates whether the set of values is missing a value for each of the plurality of points in time;
determining a set of one or more transformation weights based on the missingness pattern; and
transforming the input based on the determined transformation weights.

2. The method of claim 1, wherein determining a set of transformation weights based on the missingness pattern comprises utilizing a set of one or more attention weights.

3. The method of claim 2, wherein utilizing a set of attention weights comprises determining the set of attention weights using a set of one or more neural networks.

4. The method of claim 1, wherein the missingness pattern is a binary vector.

5. The method of claim 1, wherein transforming the input is further based on an offset that depends on the missingness pattern.

6. The method of claim 1, wherein determining the set of transformation weights comprises scaling the set of transformation weights.

7. The method of claim 1, wherein transforming the input comprises linearly mixing the missingness pattern with a fixed linear transformation.

8. The method of claim 1 further comprising updating a model based on the transformed input.

9. The method of claim 1 further comprising generating a predictive output based on the transformed input.

10. A non-transitory machine readable medium containing processor instructions for generating representations of inputs with missing values, where execution of the instructions by a processor causes the processor to perform a process that comprises:

receiving an input comprising a set of one or more values for a plurality of features, wherein the set of values for at least one of the plurality of features comprises values for each of a plurality of points in time;
identifying a missingness pattern of the input, wherein the missingness pattern for the at least one feature indicates whether the set of values is missing a value for each of the plurality of points in time;
determining a set of one or more transformation weights based on the missingness pattern; and
transforming the input based on the determined transformation weights.

11. The non-transitory machine readable medium of claim 10, wherein determining a set of transformation weights based on the missingness pattern comprises utilizing a set of one or more attention weights.

12. The non-transitory machine readable medium of claim 11, wherein utilizing a set of attention weights comprises determining the set of attention weights using a set of one or more neural networks.

13. The non-transitory machine readable medium of claim 10, wherein the missingness pattern is a binary vector.

14. The non-transitory machine readable medium of claim 10, wherein transforming the input is further based on an offset that depends on the missingness pattern.

15. The non-transitory machine readable medium of claim 10, wherein determining the set of transformation weights comprises scaling the set of transformation weights.

16. The non-transitory machine readable medium of claim 10, wherein transforming the input comprises linearly mixing the missingness pattern with a fixed linear transformation.

17. The non-transitory machine readable medium of claim 10, wherein the process further comprises updating a model based on the transformed input.

18. The non-transitory machine readable medium of claim 10, wherein the process further comprises generating a predictive output based on the transformed input.

Referenced Cited
U.S. Patent Documents
7634464 December 15, 2009 Chen
8150629 April 3, 2012 Geerts et al.
9495582 November 15, 2016 Guissin
10398389 September 3, 2019 D'Alessandro et al.
10650929 May 12, 2020 Beck et al.
10726954 July 28, 2020 Su et al.
11120528 September 14, 2021 Seely et al.
11196656 December 7, 2021 Jain et al.
11200511 December 14, 2021 London
11501429 November 15, 2022 Stamatoyannopoulos
11574462 February 7, 2023 Bhatia et al.
11610126 March 21, 2023 Heckerman et al.
11636309 April 25, 2023 Fisher et al.
20040059696 March 25, 2004 Kropaczek et al.
20040193019 September 30, 2004 Wei
20050047646 March 3, 2005 Jojic et al.
20050080462 April 14, 2005 Jenkins et al.
20080082359 April 3, 2008 Jung et al.
20090326976 December 31, 2009 Morris
20100235310 September 16, 2010 Gage et al.
20100254973 October 7, 2010 Mohapatra et al.
20110116522 May 19, 2011 Riggs et al.
20110218817 September 8, 2011 Spiegel
20130311968 November 21, 2013 Sharma
20140019059 January 16, 2014 Shankle et al.
20140046683 February 13, 2014 Michelson et al.
20140257128 September 11, 2014 Moxon et al.
20150010610 January 8, 2015 Tom et al.
20150185716 July 2, 2015 Wichmann et al.
20160140300 May 19, 2016 Purdie et al.
20160180053 June 23, 2016 Fuertinger et al.
20160217384 July 28, 2016 Leonard et al.
20160222448 August 4, 2016 Horvath
20170091615 March 30, 2017 Liu et al.
20170161635 June 8, 2017 Oono et al.
20170255871 September 7, 2017 Macready et al.
20170286627 October 5, 2017 Barhak
20170344706 November 30, 2017 Torres et al.
20170357844 December 14, 2017 Comaniciu et al.
20170364043 December 21, 2017 Ganti et al.
20170364803 December 21, 2017 Calmon et al.
20170372193 December 28, 2017 Mailhe et al.
20180018590 January 18, 2018 Szeto et al.
20180046780 February 15, 2018 Graiver et al.
20180081914 March 22, 2018 Zoll et al.
20180082172 March 22, 2018 Patel et al.
20180204138 July 19, 2018 Nugent
20180299152 October 18, 2018 Libal et al.
20180314573 November 1, 2018 Chang et al.
20180315505 November 1, 2018 Itu et al.
20180316781 November 1, 2018 Salem
20180336319 November 22, 2018 Itu et al.
20190019570 January 17, 2019 Fuertinger et al.
20190130281 May 2, 2019 Yang et al.
20190220733 July 18, 2019 Fisher et al.
20190244680 August 8, 2019 Rolfe et al.
20190283247 September 19, 2019 Chang et al.
20190303471 October 3, 2019 Lee et al.
20190303798 October 3, 2019 Xie et al.
20200035362 January 30, 2020 Abou Shousha et al.
20200143498 May 7, 2020 Alkan et al.
20200357490 November 12, 2020 Kartoun et al.
20200395103 December 17, 2020 Ramakrishnan et al.
20200401916 December 24, 2020 Rolfe et al.
20200411199 December 31, 2020 Shrager et al.
20210057108 February 25, 2021 Fisher et al.
20210090694 March 25, 2021 Colley et al.
20210117842 April 22, 2021 Smith et al.
20210158906 May 27, 2021 Xie et al.
20210241139 August 5, 2021 Jain et al.
20210241860 August 5, 2021 Bhattacharya et al.
20210256453 August 19, 2021 Morgan et al.
20210353203 November 18, 2021 Burman et al.
20220003894 January 6, 2022 Shapiro et al.
20220051796 February 17, 2022 Zhu et al.
20220121955 April 21, 2022 Chavoshi et al.
20220157413 May 19, 2022 Fisher et al.
20220172085 June 2, 2022 Fisher et al.
20220187776 June 16, 2022 Bagne
20220188601 June 16, 2022 Adler et al.
20220300329 September 22, 2022 Abhishek
20220318689 October 6, 2022 Li-Bland et al.
20220344009 October 27, 2022 Schuler da Costa Ferro
20220410385 December 29, 2022 Shimura
20220415454 December 29, 2022 Schuler da Costa Ferro et al.
20230004796 January 5, 2023 Mayer
20230118864 April 20, 2023 Zhang et al.
20230209035 June 29, 2023 Kaabi et al.
20230245258 August 3, 2023 Ma et al.
20230245777 August 3, 2023 Foschini et al.
Foreign Patent Documents
3088204 July 2019 CA
111758108 October 2020 CN
112863622 May 2021 CN
113724806 November 2021 CN
3740908 November 2020 EP
4018394 June 2022 EP
4220650 August 2023 EP
4256418 October 2023 EP
2021511584 May 2021 JP
202231730 February 2022 JP
2022544859 October 2022 JP
7305656 June 2023 JP
0229556 April 2002 WO
2006084196 August 2006 WO
2007022020 February 2007 WO
2007022020 June 2007 WO
2014114295 July 2014 WO
2016145379 September 2016 WO
2019143737 July 2019 WO
2020115487 June 2020 WO
2020154573 July 2020 WO
2021041128 March 2021 WO
2021077097 April 2021 WO
2022003528 January 2022 WO
2022101809 May 2022 WO
2022120350 June 2022 WO
2022125806 June 2022 WO
2022120350 August 2022 WO
2022187064 September 2022 WO
2022210425 October 2022 WO
2022261420 December 2022 WO
2022272308 December 2022 WO
2019143737 March 2023 WO
Other references
  • Extended European Search Report for European Application No. 19741291.9, Search completed Sep. 8, 2021, dated Sep. 17, 2021, 12 Pgs.
  • Extended European Search Report for European Application No. 23154548.4, Search completed Jun. 15, 2023, dated Jun. 23, 2023, 11 Pgs.
  • International Preliminary Report on Patentability for International Application PCT/US2019/013870 Report issued Jul. 21, 2020, dated Jul. 30, 2020, 4 Pgs.
  • International Preliminary Report on Patentability for International Application PCT/US2020/047054, Report issued Feb. 17, 2022, dated Mar. 3, 2022, 5 Pgs.
  • International Preliminary Report on Patentability for International Application PCT/US2020/056354, Report issued Apr. 19, 2022, dated Apr. 28, 2022, 05 Pgs.
  • International Preliminary Report on Patentability for International Application PCT/US2021/072678, Report issued May 30, 2023, dated Jun. 15, 2023, 08 Pgs.
  • International Search Report and Written Opinion for International Application No. PCT/US2019/013870, Search completed Mar. 18, 2019, dated Mar. 27, 2019, 9 Pgs.
  • International Search Report and Written Opinion for International Application No. PCT/US2020/047054, Search completed Oct. 8, 2020, dated Nov. 23, 2020, 10 Pgs.
  • International Search Report and Written Opinion for International Application No. PCT/US2020/056354, Search completed Dec. 21, 2020, dated Jan. 25, 2021, 09 pgs.
  • International Search Report and Written Opinion for International Application No. PCT/US2022/073165, Search completed Aug. 14, 2022, dated Sep. 7, 2022, 9 pgs.
  • International Search Report and Written Opinion for International Application PCT/US2021/072678, search completed Jan. 31, 2022, dated Jul. 1, 2022, 13 Pgs.
  • “PROCOVA™ Handbook for the Target Trial Statistician”, Ver. 1.0, European Medicines Agency, (2021).
  • Akhtar et al., “Improving the Robustness of Neural Networks Using K-Support Norm Based Adversarial Training”, IEEE Access; Publication [online]. Dec. 28, 2016, pp. 1-9.
  • Allison, “Handling Missing Data by Maximum Likelihood”, SAS Global Forum 2012: Statistics and Data Analysis, 21 pages.
  • Arici et al., “Associative Adversarial Networks”, arXiv:1611.06953v1 [cs.LG], Nov. 18, 2016, 8 pgs. <URL:https://arxiv.org/abs/1611. 06953>.
  • Arjovsky et al., “Wasserstein GAN”, arXiv: 1701.07875v1 [stat.ML], Jan. 26, 2017, 30 pgs.
  • Balzer et al., “Adaptive pair-matching in randomized trials with unbiased and efficient effect estimation”, Statist. Med. 2015, 34 999-1011; DOI: 10.1002/sim.6380.
  • Bengio et al., “Greedy Layer-Wise Training of Deep Networks”, Advances in Neural Information Processing Systems, 2007, 13 pgs.
  • Burges et al., “Learning to Rank Using Gradient Descent”, Proceedings of the 22nd International Conference on Machine Learning, 2005, 8 pages.
  • Chatterjee et al., “Explaining Complex Distributions with Simple Models”, 2008. Econophysics. pp. 1-15.
  • Cho et al., “Gaussian-Bernoulli deep Boltzmann machine”, Proceedings of the 2013 International Joint Conference on Neural Networks (IJCNN), Dallas, Texas, Aug. 4-9, 2013, 9 pgs.
  • Coon et al., “A High-Density Whole-Genome Association Study Reveals That APOE Is the Major Susceptibility Gene for Sporadic Late-Onset Alzheimer's Disease”, The Journal of Clinical Psychiatry, 2007, vol. 68, No. 04, pp. 613-618, 2007, doi: 10.4088/jcp.v68n0419.
  • Cui et al., “Multilevel Modeling and Value of Information in Clinical Trial Decision Support”, BMC Systems Biology (2014) 8:6; DOI 10.1186/s12918-014-0140-0.
  • Davidian et al., “Semiparametric Estimation of Treatment Effect in a Pretest-Posttest Study”, Biometrics, Dec. 2003, vol. 59, No. 4, pp. 1046-1055, 2003. ISSN 0006-341X. doi: 10.1111/ j.0006-341x.2003.00120.x.
  • Dutt et al., “Generative Adversarial Networks (GAN) Review”, CVR Journal of Science and Technology, Dec. 2017, vol. 13, pp. 1-5.
  • Eickhoff et al., “Copulas for Information Retrieval”, Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, Jul. 2013, pp. 663-672.
  • Fisher et al., “Boltzmann Encoded Adversarial Machines”, arxiv.org:1804.08682v1, Apr. 23, 2018, XP081229135, 17 pgs.
  • Gabrie et al., “Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy”, Advances in Neural Information Processing Systems, vol. 28, 2015, 9 pgs.
  • Goodfellow et al., “Generative Adversarial Nets”, Advances in Neural Information Processing Systems, vol. 27, 2014, 9 pgs.
  • Graham et al., “Analysis with missing data in drug prevention research”, NIDA Research Monograph, Feb. 1994, vol. 142, pp. 13-63.
  • Greydanus, “Generative Adversarial Networks for the MNIST dataset”, “Mnist gan,” http://github.com/greydanus/mnist-gan (2017), 2 pgs.
  • Grover et al., “Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models”, arXiv:1705.08868, May 24, 2017, 10 pages.
  • Gupta, “Intention-to-treat concept: a review”, Perspectives in Clinical Research, Jul. 2011, vol. 2, No. 3, pp. 109-112, doi: 10.4103/2229-3485.83221.
  • Hannan, “Randomized Clinical Trials and Observational Studies: Guidelines for Assessing Respective Strengths and Limitations”, JACC: Cardiovascular Interventions, Jun. 2008, vol. 1, No. 3, pp. 211-217, https://doi.org/10.1016/j.jcin.2008.01.008.
  • Hinton, “A Practical Guide to Training Restricted Boltzmann Machines”, University of Toronto, Technical Report UTML TR 2010-003, 2010, 21 pages. <URL: http://www.cs.toronto.edu/ hinton/absps/guideTR. pdf>.
  • Hinton et al., “A Fast Learning Algorithm for Deep Belief Nets”, Neural Computation 18, 1527-1554 (2006).
  • Hinton et al., “Reducing the Dimensionality of Data with Neural Networks”, Science, vol. 313, No. 5786, Jul. 28, 2006, pp. 504-507.
  • Karcher et al., “The “RCT augmentation”: a novel simulation method to add patient heterogeneity into phase III trials”, BMC Medical Research Methodology (2018) 18:75; https://doi.org/10.1186/s1287 4-018-0534-6.
  • Kim et al., “Deep Directed Generative Models with Energy-Based Probability Estimation”, arXiv:1606.03439, Jun. 10, 2016, 9 pages. <URL: https : //arxiv. org/abs/1606. 03439>.
  • Kullback et al., “On Information and Sufficiency”, The Annals of Mathematical Statistics, vol. 22, No. 1, 1951, pp. 79-86.
  • Lamb et al., “GibbsNet: Iterative Adversarial Inference for Deep Graphical Models”, arXiv preprint arXiv:1712.04120v1, 2017, 11 pages.
  • Li et al., “Temperature based Restricted Boltzmann Machines”, Scientific Reports Jan. 13, 2016, vol. 19133, 12 pgs., DOI:10.1038/srep19133.
  • Liu et al., “A Survey of Deep Neural Network Architectures and their Applications”, Neurocomputing, vol. 234, Apr. 19, 2017, pp. 11-26.
  • Liu et al., “Image inpainting for irregular holes using partial convolutions”, Proceedings of the European conference on computer vision (ECCV), 2018, pp. 85-100.
  • Lopez-Ruiz et al., “Equiprobability, Entropy, Gamma Distributions and Other Geometrical Questions in Multi-Agent Systems”, Entropy, 2009, vol. 11, pp. 959-971, doi:10.3390/e11040959.
  • Maldonado, “Estimating causal effects”, International Journal of Epidemiology, Apr. 2002, vol. 31, No. 2, pp. 422-429.
  • Marlin et al., “Recommender systems: Missing data and statistical model estimation”, Proceedings of the Twenty-Second international joint conference on Artificial Intelligence, 2011, vol. Three (IJCAI'11), AAAI Press, pp. 2686-2691.
  • Miotto et al., “Deep Learning for Healthcare: Review, Opportunities and Challenges”, Briefings in Bioinformatics , vol. 19, No. 6, 2017, 11 pgs.
  • Montavon et al., “Wasserstein Training of Restricted Boltzmann Machines”, Advances in Neural Information Processing Systems, vol. 29, 2016, 9 pgs.
  • Neville et al., “Development of a unified clinical trial database for Alzheimer's disease”, Alzheimer's & Dementia: The Journal of the Alzheimer's Association, 2015, vol. 11, No. 10, pp. 1212-1221, https://doi.org/10.1016/j.jalz.2014.11.005.
  • Nguyen et al., “Latent Patient Profile Modelling and Applications with Mixed-Variate Restricted Boltzmann Machine”, Advances in Knowledge Discovery and Data Mining, 2013, pp. 123-135.
  • Nguyen et al., “Supervised Restricted Boltzmann Machines”, UAI. 2017, 10 pgs.
  • Niu et al., “A review on the attention mechanism of deep learning”, Neurocomputing, vol. 452, 2021, pp. 48-62, https://doi.org/10.1016/j.neucom.2021.03.091.
  • Overhage et al., “Desideratum for evidence based epidemiology”, Drug Safety, 2013, vol. 36, Suppl. 1, pp. 5-14, DOI: 10.1007/s40264-013-0102-2.
  • Quinn et al., “Docosahexaenoic Acid Supplementation and Cognitive Decline in Alzheimer Disease: A Randomized Trial”, Journal of the American Medical Association, Nov. 3, 2010, vol. 304, No. 17, pp. 1903-1911, doi: 10.1001/jama.2010.1510.
  • Rogers et al., “Combining patient-level and summary-level data for Alzheimer's disease modeling and simulation: a beta regression meta-analysis”, Journal of Pharmacokinetics and Pharmacodynamics 39.5 (2012), 20 pgs.
  • Romano et al., “Resurrecting weighted least squares”, Journal of Econometrics, 197(1), 48 pgs., Available at: https://doi.org/10.1016/j.jeconom.2016.10.003, (2016).
  • Romero et al., “The coali-tion against major diseases: developing tools for an integrated drug development process for alzheimer's and parkinson's diseases”, Clinical Pharmacology & Therapeutics, Aug. 12, 2009, vol. 86, No. 4, pp. 365-367, https://doi.org/10.1038/clpt.2009.165.
  • Rosen et al., “A New Rating Scale for Alzheimer's Disease”, American Journal of Psychiatry, Nov. 1984, vol. 141, Issue 11, pp. 1356-1364, https://doi.org/10.1176/ajp.141.11.1356.
  • Royston, “A Combined Test for a Generalized Treatment Effect in Clinical Trials with a Time-to-Event Outcome”, The Stata Journal, 2017, 17, No. 2, pp. 405-421.
  • Royston et al., “A Simulation Study Comparing the Power of Nine Tests of the Treatment Effect in Randomized Controlled Trials with a Time-to-Event Outcome”, Royston and Parmar Trials (2020). Retrieved on Aug. 14, 2022. Retrieved from <URL: https://link.springer.com/content/pdf/10.1186/s13063-020-4153-2.pdf> entire document.
  • Royston et al., “Augmenting the Logrank Test in the Design of Clinical Trials in which Non-Proportional Hazards of the Treatment Effect may be Anticipated”, Royston and Parmar BMC Medical Research Methodology (2016). Retrieved on Aug. 14, 2022. Retrieved from <URL:https://bmcmedresmethodol.biomedcentral.conn/track/pdf/10.1186/s12874-016-0110-x.pdf> entire document.
  • Rubin, “Causal Inference Using Potential Outcomes: Design, Modeling, Decisions”, Journal of the American Statistical Association, Mar. 2005, vol. 100, No. 469, pp. 322-331, https://doi.org/10.1198/016214504000001880.
  • Salakhutdinov et al., “Deep Boltzmann Machines”, Proc. International Conference on Artificial Intelligence and Statistics, 2009, pp. 448-455.
  • Schuler et al., “Increasing the efficiency of randomized trial estimates via Linear Adjustment for a prognostic score”, The International Journal of Biostatistics, 18(2), pp. 329-356. Available at: https://doi.org/10.1515/ijb-2021-0072, (2021).
  • Silva et al., “Predicting In-Hospital Mortality of ICU Patients: The PhysioNet/Computing in Cardiology Challenge 2012”, Computing in Cardiology, 2012, vol. 39, pp. 245-248.
  • Song et al., “Generative Adversarial Learning of Markov Chains”, Accessed at URL https://openreview.net/forum?id=S1L-hCNtl, 2017, 7 Pages.
  • Sox et al., “The Methods of Comparative Effectiveness Research”, Annual Review of Public Health, Apr. 2012, vol. 33, pp. 425-445, doi: 10.1146/annurev-publhealth-031811-124610.
  • Sterne et al., “Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls”, BMJ, 2009, pp. 338, doi: https://doi.org/10.1136/bmj.b2393 (Published Jun. 29, 2009).
  • Sutskever, et al., “The Recurrent Temporal Restricted Boltzmann Machine”, Advances in Neural Information Processing Systems, 2009, pp. 1601-1608.
  • Tan et al., “A tensor-based method for missing traffic data completion”, Transportation Research Part C: Emerging Technologies, vol. 28, 2013, pp. 15-27, https://doi.org/10.1016/j.trc.2012.12.007.
  • Tipirneni et al., “Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series”, ACM Transactions on Knowledge Discovery from Data (TKDD), 2022, vol. 16, No. 6, pp. 1-17.
  • Tran et al., “Mixed-Variate Restricted Boltzmann Machines”, Asian Conference on Machine Learning, JMLR: Workshop and Conference Proceedings 20, 2011, pp. 213-229.
  • Tuzman, “Broadening role for external control arms in clinical trials”, Biocentury, Tools & Techniques, reprint from Jul. 15, 2019, 5 pgs.
  • Ventz et al., “Design and Evaluation of an External Control Arm Using Prior Clinical Trials and Real-World Data”, Clincal Cancer Resarch 2019; 25:4993-5001; doi: 10.1158/1078-0432.CCR-19-0820.
  • Yi et al., “ST-MVL: Filling Missing Values in Geo-Sensory Time Series Data”, Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, 2016, pp. 2704-2710.
  • Yu et al., “Assessment and adjustment of approximate inference algorithms using the law of total variance”, arXiv preprint arXiv:1911.08725, Nov. 2019 [online], [retrieved on Jan. 31, 2022].
  • Zhang et al., “Predictive Deep Boltzmann Machine for Multiperiod Wind Speed Forecasting”, IEEE Transactions on Sustainable Energy, 2015, vol. 6, Issue 4, pp. 1416-1425, doi: 10.31109/TSTE.2015.244387.
  • Extended European Search Report for European Application No. 20857028.3, Search completed Aug. 11, 2023, dated Aug. 21, 2023, 12 Pgs.
  • International Search Report and Written Opinion for International Application No. PCT/US2023/069016, Search completed Sep. 18, 2023, dated Oct. 18, 2023, 11 Pgs.
  • Duan et al., “Utilizing dynamic treatment information for MACE prediction of acute coronary syndrome”, 2019, BMC Medical Informatics and Decision Making, pp. 1-11 (Year: 2019).
  • Fisher et al., “Machine Learning for Comprehensive Forecasting of Alzheimer's Disease Progression”, Sep. 20, 2019, Scientific Reports, pp. 1-41. (Year: 2019).
  • Herlau et al., “Bayesian Dropout”, arXiv e-prints (2015): arXiv-1508.
  • Jerez et al., “Missing data imputation using statistical and machine learning methods in a real breast cancer problem”, Artificial Intelligence in Medicine, (Year: 2010), vol. 50, Issue 2.
  • Lim et al., “Time-Series Forecasting with Deep Learning: A Survey”, Jul. 28, 2021, The Royal Society Publishing, pp. 1-14 (Year: 2021).
  • Lipton et al., “Modeling missing data in clinical time series with rnns”, Machine Learning for Healthcare. Jun. 13, 2016;56(56):253-70.
  • Shan et al., “Accurate Unconditional p-Values for a Two-Arm Study with Binary Endpoints”, Journal of Statistical Computation and Simulation, [Online] Apr. 13, 2018, vol. 88, No. 6, pp. 1200-1210, XP093073190, ISSN: 0094-9655, DOI: 10.1080/00949655.2018.1425690, Retrieved from the Internet: URL:https://www.ncbi.nlm.nih.gov/pmc/artic les/PMC6510515/pdf/nihms-1504617.pdf> [retrieved on Jan. 28, 2018].
  • Shan et al., “Exact p-Values for Simon's Two-Stage Designs in Clinical Trials”, Statistics in Biosciences, Springer US,Boston, [Online]Jun. 16, 2016, vol. 8, No. 2, pp. 351-357, XP036062971,ISSN: 1867-1764, DOI:10.1007/S12561-016-9152-1, Retrieved from the Internet: URL:https://www.ncbi.nlm.nih.gov/pmc/artic les/PMC5167475/>.
  • Zhu et al., “Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data”, arXiv preprint arXiv:1901.06314 (2019).
Patent History
Patent number: 11868900
Type: Grant
Filed: Jun 9, 2023
Date of Patent: Jan 9, 2024
Assignee: Unlearn.AI, Inc. (San Francisco, CA)
Inventors: Aaron Michael Smith (Corte Madera, CA), Charles Kenneth Fisher (Truckee, CA), Franklin D. Fuller (San Francisco, CA)
Primary Examiner: Juan A Torres
Application Number: 18/332,631
Classifications
Current U.S. Class: Neural Network (706/15)
International Classification: G06N 3/08 (20230101); G06N 3/096 (20230101);