MACHINE LEARNING MODELS FOR DETECTING OUTLIERS AND ERRONEOUS SENSOR USE CONDITIONS AND CORRECTING, BLANKING, OR TERMINATING GLUCOSE SENSORS
Methods, systems, and devices for improving continuous glucose monitoring (“CGM”) are described herein. More particularly, the methods, systems, and devices describe retrieving a machine learning model that is trained to classify CGM sensor data and blanking the CGM sensor data based on an outlier classification from the machine learning model. The system may terminate sensors for which there is an aggregation of blanked CGM sensor data. The methods, systems, and devices described herein may additionally comprise a machine learning model that is trained to detect and correct for erroneous sensor use conditions based on error patterns in sensor data. The system may determine resolutions for correcting the detected erroneous sensor use conditions.
The present technology is generally related to sensor technology, including sensors used for sensing a variety of physiological parameters, e.g., glucose concentration.
BACKGROUNDOver the years, a variety of sensors have been developed for detecting and/or quantifying specific agents or compositions in a patient's blood, which enable patients and medical personnel to monitor physiological conditions within the patient's body. Illustratively, subjects may wish to monitor blood glucose levels in a subject's body on a continuing basis. Thus, glucose sensors have been developed for use in obtaining an indication of blood glucose levels in a diabetic patient. Such readings are useful in monitoring and/or adjusting a treatment regimen which typically includes the regular administration of insulin to the patient. Presently, a patient can measure his/her blood glucose (“BG”) using a BG measurement device (i.e., glucose meter), such as a test strip meter, a continuous glucose measurement system (or a continuous glucose monitor), or a hospital BG test. BG measurement devices use various methods to measure the BG level of a patient, such as a sample of the patient's blood, a sensor in contact with a bodily fluid, an optical sensor, an enzymatic sensor, or a fluorescent sensor. When the BG measurement device has generated a BG measurement, the measurement is displayed on the BG measurement device.
SUMMARYCurrent continuous glucose monitoring (“CGM”) systems use CGM calibration algorithms to determine when measurements are accurate. For example, accuracy may change based on wear, battery life, and other factors. Current CGM systems determine accuracy based on an independent input signal, whereas systems and methods described herein utilize a multi-dimensional input signal for determining accuracy. The multi-dimensional input signal may include an Interstitial Current Signal (“Isig”), an Electrochemical Impedance Spectroscopy Signal (“EIS”), a counter voltage (“Vcntr”), and/or other signals. The multi-dimensional input signal improves CGM performance, leading to more reliable determinations of accuracy, wear, battery life, and other factors and providing a user with more accurate data.
Though the use of a multi-dimensional input signal improves CGM performance, present CGM requirements are difficult to scale with the multi-dimensional inputs. Government agencies (e.g., the Federal Drug Administration (“FDA”)) impose restrictions and requirements for the sensitivity and accuracy of CGMs. For example, CGM devices are required to meet numerous criteria (e.g., FDA's integrated continuous glucose monitoring (“iCGM”) criteria) in order for the sensor data to be considered accurate enough to qualify for preferential treatment during regulatory the regulatory approval process. The iCGM criteria is designed to characterize the relative distribution of error of a CGM system to balance the often-competing needs of an overall mean error value and short tails in the error distribution. In addition, current systems used as predicate devices in fashioning the iCGM criteria are based on single signal analysis, so the criteria may be difficult to generalize to systems based on multi-dimensional input signals as described herein. To solve these problems, methods, systems, and devices described herein may train a machine learning model to classify multi-dimensional input signals in accordance with the iCGM criteria. The outputs from the trained machine learning model may be used to blank (i.e., remove, ignore) measurements, during computation within a glucose estimation device, which do not meet the iCGM criteria. Thus, the methods, systems, and devices described herein allow for improved CGM techniques that are compatible with the FDA's iCGM criteria and related criteria designed to balance gross measures of accuracy with error distributions featuring shorter tail.
More particularly, the methods, systems, and devices describe training a machine learning model to identify outlier measurements based on behavior signatures and informed by the iCGM and similar criteria. The machine learning model may take as inputs multi-dimensional CGM sensor data and may use training data to set model parameters. The training data may include clinical data on iCGM performance. In some embodiments, the system may classify the training data according to known classifications (e.g., large negative bias, large positive bias, nominal accuracy, poor accuracy, intermediate accuracy, good accuracy, or other classifications). The system may receive multi-dimensional CGM sensor data from a sensor electrode or another computing device and may input the sensor data into the machine learning model it contains or contained in another computing device. Outputs from the machine learning model may indicate whether the CGM sensor data corresponds to an outlier (e.g., or to a known classification, as described above) or fits a known pattern associated with an error state or error condition. If the CGM sensor data corresponds to an outlier, the system may not process the corresponding sensor data or may blank or not display the sensor data on a display interface of a user's device. In some examples, the system may blank sensor data by foregoing to transmit the sensor data to a user device or any other device with a display interface.
Another limitation of current systems is a loss of data due to blanking. For example, if a sensor device is consistently blanking, the quantity of data processed by the device or received by a user of the sensor device may be inadequate. Accordingly, methods, systems, and devices described herein may terminate the sensor device (e.g., stop transmitting sensor data from that sensor device) based on determining that a certain number of outlier measurements has been reached or a certain number of blanking instances has been activated during a specified period during sensor wear. More particularly, the methods, systems, and devices describe using an outlier counter (e.g., or a buffer, threshold, or any other means) to track outlier measurements. For example, the system may reset the outlier counter (e.g., to zero) upon determining that a sensor datapoint does not correspond to an outlier measurement (e.g., as determined by the machine learning model). The system may increment the outlier counter for each datapoint that is classified as an outlier by the machine learning model. The system may use various additional criteria in conjunction with the outlier counter (e.g., or other means for tracking outlier measurements). For example, the outlier counter may track a number of outlier measurements that are received consecutively, in a certain time period, or within any other constraints. Once the outlier counter breaches (e.g., meets or exceeds) a threshold, the system may terminate the sensor device. In some embodiments, the system may alert the user that the sensor device must be replaced.
In some aspects, methods, systems, and devices for continuous glucose monitoring are described. For example, the system may retrieve a machine learning model that is trained to identify outlier measurements based on iCGM criteria using training data. In some embodiments, the training data may include clinical data on iCGM performance. The system may receive CGM sensor data and input the sensor data into the machine learning model. The system may receive an output from the machine learning model indicating that the sensor data corresponds to an outlier measurement. Based on the output, the system may blank (e.g., ignore, eschew from transmitting to another device) or not display the sensor data (e.g., on a display interface) on a user device.
Another limitation of conventional continuous glucose monitoring (“CGM”) systems is inability to dynamically detect and correct errors in the sensor data. CGM systems continuously present sensor data for a user of a sensor device. The system must therefore process and respond to characteristics of the sensor data in real-time. Notably, any error in this data may result in severe or even fatal effects to the user. Moreover, conventional CGM systems lack the functionality to detect and correct for errors in the data as they arise. Systems and methods described herein train a machine learning model to detect error patterns in the sensor data. For example, the system may train the machine learning model to recognize error pattern characteristics and identify underlying erroneous sensor use conditions. However, conventional pattern recognition technology would be unsuccessful at identifying erroneous sensor use conditions associated with error patterns in the sensor data, as pattern recognition alone is insufficient for understanding the complexities of the sensor data surrounding the error patterns. Systems and methods described herein thus rely upon context information relating to the sensor data in order to determine the erroneous sensor use condition. For example, the system relies upon historic information relating to the sensor data, such as behaviors, trends, and patterns of the sensor data within a time period. The behaviors, trends, and patterns of the sensor data allow the system to identify between identical error patterns that are associated with different erroneous sensor use conditions. Additionally, the system accounts for behaviors and patterns of the various sensor input features that make up the sensor data. Identifying the correct erroneous sensor use condition associated with the detected error pattern is essential for correcting the error. Once the system has identified the correct erroneous sensor use condition, the system may determine a viable resolution for correcting the error in the data.
In order to comply with the iCGM criteria, the CGM system must ensure that sensor data which does not comply with the iCGM criteria is not shown to the user. With current systems, this leads to excessive blanking in response to abnormal sensor data. Such excessive blanking may deprive a user of a sensor device of valuable sensor glucose data. Systems and methods described herein improve upon current systems by detecting and correcting for errors in complex sensor data in real time in order to maximize the accurate data that the system is able to provide to the user in compliance with iCGM criteria.
More particularly, methods, systems, and devices for continuous glucose monitoring are described. For example, the system may retrieve a machine learning model that is trained to identify erroneous sensor use conditions based on sensor data error patterns using training data. In some embodiments, the training data may include clinical data on erroneous sensor use conditions. The system may receive CGM sensor data and input the sensor data into the machine learning model. The system may receive an output from the machine learning model indicating an erroneous sensor use condition based on an error pattern identified in the sensor data. Based on the output, the system may determine and implement a resolution in order to correct the identified erroneous sensor use condition.
Various other aspects, features, and advantages will be apparent through the detailed description and the drawings attached hereto. It is also to be understood that both the foregoing general description and the following detailed description are examples and not restrictive of the scope of the invention. As used in the specification and in the claims, the singular forms of “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. In addition, as used in the specification and the claims, the term “or” means “and/or” unless the context clearly dictates otherwise. Additionally, as used in the specification “a portion,” refers to a sub-part of, or the entirety of, a given item (e.g., data) unless the context clearly dictates otherwise.
The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.
A detailed description of embodiments of the invention will be made with reference to the accompanying drawings, wherein like numerals designate corresponding parts in the figures.
In the following description, reference is made to the accompanying drawings which form a part hereof and which illustrate several embodiments of the present inventions. It is understood that other embodiments may be utilized, and structural and operational changes may be made without departing from the scope of the present inventions.
The inventions herein are described below with reference to flowchart illustrations of methods, systems, devices, apparatus, and programming and computer program products. It will be understood that each block of the flowchart illustrations, and combinations of blocks in the flowchart illustrations, can be implemented by programing instructions, including computer program instructions (as can any menu screens described in the figures). These computer program instructions may be loaded onto a computer or other programmable data processing apparatus (such as a controller, microcontroller, or processor in a sensor electronics device) to produce a machine, such that the instructions which execute on the computer or other programmable data processing apparatus create instructions for implementing the functions specified in the flowchart block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks, and/or menus presented herein. Programming instructions may also be stored in and/or implemented via electronic circuitry (e.g., storage circuitry, processing circuitry), including integrated circuits (ICs) and Application Specific Integrated Circuits (ASICs) used in conjunction with sensor devices, apparatuses, and systems. The following terms and definitions may also be used herein:
In particular embodiments, the subcutaneous sensor set 10 facilitates accurate placement of a flexible thin film electrochemical sensor 12 of the type used for monitoring specific blood parameters representative of a user's condition. The sensor 12 monitors glucose levels in the body and may be used in conjunction with automated or semi-automated medication infusion pumps (e.g., wearable sensor electronics device 100, as shown in
Particular embodiments of the flexible electrochemical sensor 12 are constructed in accordance with thin film mask techniques to include elongated thin film conductors embedded or encased between layers of a selected insulative material such as polyimide film or sheet, and membranes. The sensor electrodes 20 at a tip end of the sensing portion 18 are exposed through one of the insulative layers for direct contact with patient blood or other body fluids, when the sensing portion 18 (or active portion) of the sensor 12 is subcutaneously placed at an insertion site. The sensing portion 18 is joined to a connection portion 24 that terminates in conductive contact pads, or the like, which are also exposed through one of the insulative layers. In alternative embodiments, other types of implantable sensors, such as chemical based, optical based, or the like, may be used.
As is known in the art, the connection portion 24 and the contact pads are generally adapted for a direct wired electrical connection to a suitable monitor or sensor electronics device 200 (e.g., wearable sensor electronics devices 100 or 150, as shown in
The sensor electrodes 20 may be used in a variety of sensing applications and may be configured in a variety of ways. For example, the sensor electrodes 20 may be used in physiological parameter sensing applications in which some type of biomolecule is used as a catalytic agent. For example, the sensor electrodes 20 may be used in a glucose and oxygen sensor having a glucose oxidase (GOx) enzyme catalyzing a reaction with the sensor electrodes 20. The sensor electrodes 20, along with a biomolecule or some other catalytic agent, may be placed in a human body in a vascular or non-vascular environment. For example, the sensor electrodes 20 and biomolecule may be placed in a vein and be subjected to a blood stream or may be placed in a subcutaneous or peritoneal region of the human body.
The monitor 200 may also be referred to as a sensor electronics device 200. The monitor 200 may include a power source 210, a sensor interface 222, processing electronics 224, and data formatting electronics 228. The monitor 200 may be coupled to the sensor set 10 by a cable 202 through a connector that is electrically coupled to the connector block 28 of the connection portion 24. In an alternative embodiment, the cable may be omitted. In this embodiment, the monitor 200 may include an appropriate connector for direct connection to the connection portion 204 of the sensor set 10. The sensor set 10 may be modified to have the connector portion 204 positioned at a different location, e.g., on top of the sensor set to facilitate placement of the monitor 200 over the sensor set.
In one embodiment, the sensor interface 222, the processing electronics 224, and the data formatting electronics 228 are formed as separate semiconductor chips, however, alternative embodiments may combine the various semiconductor chips into a single, or multiple customized semiconductor chips. The sensor interface 222 connects with the cable 202 that is connected with the sensor set 10.
The power source 210 may be a battery. The battery can include three series silver oxide 357 battery cells. In alternative embodiments, different battery chemistries may be utilized, such as lithium-based chemistries, alkaline batteries, nickel metal hydride, or the like, and a different number of batteries may be used. The monitor 200 provides power to the sensor set via the power source 210, through the cable 202 and cable connector 204. In one embodiment, the power is a voltage provided to the sensor set 10. In another embodiment, the power is a current provided to the sensor set 10. In an embodiment, the power is a voltage provided at a specific voltage to the sensor set 10.
The sensor electrodes 410 may be used in a variety of sensing applications and may be configured in a variety of ways. For example, the sensor electrodes 410 may be used in physiological parameter sensing applications in which some type of biomolecule is used as a catalytic agent. For example, the sensor electrodes 410 may be used in a glucose and oxygen sensor having a GOx enzyme catalyzing a reaction with the sensor electrodes 410. The sensor electrodes 410, along with a biomolecule or some other catalytic agent, may be placed in a human body in a vascular or non-vascular environment. For example, the sensor electrodes 410 and biomolecule may be placed in a vein and be subjected to a blood stream.
The sensor 555 creates a sensor signal indicative of a concentration of a physiological characteristic being measured. For example, the sensor signal may be indicative of a blood glucose reading. In an embodiment utilizing subcutaneous sensors, the sensor signal may represent a level of hydrogen peroxide in a subject. In an embodiment where blood or cranial sensors are utilized, the amount of oxygen is being measured by the sensor and is represented by the sensor signal. In an embodiment utilizing implantable or long-term sensors, the sensor signal may represent a level of oxygen in the subject. The sensor signal is measured at the working electrode 575. In one embodiment, the sensor signal may be a current measured at the working electrode. In an embodiment, the sensor signal may be a voltage measured at the working electrode.
The signal processor 590 receives the sensor signal (e.g., a measured current or voltage) after the sensor signal is measured at the sensor 555 (e.g., the working electrode). The signal processor 590 processes the sensor signal and generates a processed sensor signal. The measurement processor 595 receives the processed sensor signal and calibrates the processed sensor signal utilizing reference values. In one embodiment, the reference values are stored in a reference memory and provided to the measurement processor 595. The measurement processor 595 generates sensor measurements. The sensor measurements may be stored in a measurement memory (not shown) or by circuitry (e.g., storage circuitry). The sensor measurements may be sent to a display/transmission device to be either displayed on a display in a housing with the sensor electronics or transmitted to an external device.
The sensor electronics device 560 may be a monitor which includes a display to display physiological characteristics readings. The sensor electronics device 560 may also be installed in a desktop computer, a pager, a television including communications capabilities, a laptop computer, a server, a network computer, a personal digital assistant (PDA), a portable telephone including computer functions, an infusion pump including a display (e.g., wearable sensor electronics device 100, as shown in
The microcontroller 510 includes software program code, which when executed, or programmable logic which, causes the microcontroller 510 to transmit a signal to the DAC 520, where the signal is representative of a voltage level or value that is to be applied to the sensor 555. The DAC 520 receives the signal and generates the voltage value at the level instructed by the microcontroller 510. In one embodiment, the microcontroller 510 may change the representation of the voltage level in the signal frequently or infrequently. Illustratively, the signal from the microcontroller 510 may instruct the DAC 520 to apply a first voltage value for one second and a second voltage value for two seconds.
The sensor 555 may receive the voltage level or value. In one embodiment, the counter electrode 565 may receive the output of an operational amplifier which has as inputs the reference voltage and the voltage value from the DAC 520. The application of the voltage level causes the sensor 555 to create a sensor signal indicative of a concentration of a physiological characteristic being measured. In an embodiment, the microcontroller 510 may measure the sensor signal (e.g., a current value) from the working electrode. Illustratively, a sensor signal measurement circuit 531 may measure the sensor signal. In an embodiment, the sensor signal measurement circuit 531 may include a resistor and the current may be passed through the resistor to measure the value of the sensor signal. In an embodiment, the sensor signal may be a current level signal and the sensor signal measurement circuit 531 may be a current-to-frequency (I/F) converter 530. The current-to-frequency converter 530 may measure the sensor signal in terms of a current reading, convert it to a frequency-based sensor signal, and transmit the frequency-based sensor signal to the microcontroller 510. In some embodiments, the microcontroller 510 may be able to receive frequency-based sensor signals easier than non-frequency-based sensor signals. The microcontroller 510 receives the sensor signal, whether frequency-based or non-frequency-based, and determines a value for the physiological characteristic of a subject, such as a blood glucose level. The microcontroller 510 may include program code, which when executed or run, is able to receive the sensor signal and convert the sensor signal to a physiological characteristic value. In one embodiment, the microcontroller 510 may convert the sensor signal to a blood glucose level. In an embodiment, the microcontroller 510 may utilize measurements stored within an internal memory or by circuitry (e.g., storage circuitry) in order to determine the blood glucose level of the subject. In an embodiment, the microcontroller 510 may utilize measurements stored within a memory external to the microcontroller 510 or by circuitry to assist in determining the blood glucose level of the subject.
After the physiological characteristic value is determined by the microcontroller 510, the microcontroller 510 may store measurements of the physiological characteristic values for a number of time periods. For example, a blood glucose value may be sent to the microcontroller 510 from the sensor in intervals (e.g., every second or five seconds), and the microcontroller may save sensor measurements in intervals (e.g., for five minutes or ten minutes of estimated SG values from the calibration algorithm). The microcontroller 510 may transfer the measurements of the physiological characteristic values to a display on the sensor electronics device 560 or a display interface of another user device. For example, the sensor electronics device 560 may be a monitor which includes a display that provides a blood glucose reading for a subject. In one embodiment, the microcontroller 510 may transfer the measurements of the physiological characteristic values to an output interface of the microcontroller 510. The output interface of the microcontroller 510 may transfer the measurements of the physiological characteristic values, e.g., sensor glucose values, to an external device, e.g., an infusion pump (e.g., wearable sensor electronics device 100, as shown in
In a long-term sensor embodiment, where a GOx enzyme is used as a catalytic agent in a sensor, current may flow from the counter electrode 636 to a working electrode 634 only if there is oxygen in the vicinity of the enzyme and the sensor electrodes 610. Illustratively, if the voltage set at the reference electrode 632 is maintained at about 0.5 volts, the amount of current flowing from the counter electrode 636 to a working electrode 634 has a fairly linear relationship with unity slope to the amount of oxygen present in the area surrounding the enzyme and the electrodes. Thus, increased accuracy in determining an amount of oxygen in the blood may be achieved by maintaining the reference electrode 632 at about 0.5 volts and utilizing this region of the current-voltage curve for varying levels of blood oxygen. Different embodiments may utilize different sensors having biomolecules other than a glucose oxidase enzyme and may, therefore, have voltages other than 0.5 volts set at the reference electrode.
As discussed above, during initial implantation or insertion of the sensor 610, the sensor 610 may provide inaccurate readings due to the adjusting of the subject to the sensor and also electrochemical byproducts caused by the catalyst utilized in the sensor. A stabilization period is needed for many sensors in order for the sensor 610 to provide accurate readings of the physiological parameter of the subject. During the stabilization period, the sensor 610 does not provide accurate blood glucose measurements. Users and manufacturers of the sensors may desire to improve the stabilization timeframe for the sensor so that the sensors can be utilized quickly after insertion into the subject's body or a subcutaneous layer of the subject.
In previous sensor electrode systems, the stabilization period or timeframe was one hour to three hours. In order to decrease the stabilization period or timeframe and increase the timeliness of accuracy of the sensor, a sensor (or electrodes of a sensor) may be subjected to a number of pulses rather than the application of one pulse followed by the application of another voltage. for the second time period. In one embodiment, the first voltage may be 1.07 volts. In an embodiment, the first voltage may be 0.535 volts. In an embodiment, the first voltage may be approximately 0.7 volts.
At step 702, process 700 (e.g., using components described in
In some other embodiments, process 700 may determine whether a set of training sensor data in the training data corresponds to known classification, wherein the known classification comprises poor accuracy, intermediate accuracy, or good accuracy. In some embodiments, accuracy may be measured using the mean absolute relative difference (MARD) and bias. Depending on the range of the estimated SG value, a hit criteria for iCGM may shift between the MARD and bias metrics. For example, for certain SG values, the hit criteria may specify a certain bias, while for other SG values, the hit criteria may specify a MARD. In some embodiments, in order to determine whether the training sensor data has poor accuracy, intermediate accuracy, or good accuracy, process 700 may compare the training sensor data to one or more thresholds (e.g., an intermediate accuracy threshold and a good accuracy threshold). Based on whether the training sensor data satisfies the intermediate accuracy threshold, process 700 may determine whether the training sensor data has intermediate accuracy. Based on whether the training sensor data satisfies the good accuracy threshold, process 700 may determine whether the training sensor data has good accuracy. If the training sensor data does not satisfy either threshold, process 700 may determine that the training sensor data has poor accuracy. In some embodiments, process 700 may label the training sensor data with the known classification.
At step 704, process 700 (e.g., using components described in
At step 706, process 700 (e.g., using components described in
At step 708, process 700 (e.g., using components described in
At step 710, process 700 (e.g., using components described in
It is contemplated that the steps or descriptions of
At step 802, process 800 (e.g., using components described in
At step 804, process 800 (e.g., using components described in
At step 808, process 800 (e.g., using components described in
It is contemplated that the steps or descriptions of
In some embodiments, the machine learning model system may include one or more neural networks or other machine learning models. As an example, neural networks may be based on a large collection of neural units (or artificial neurons). Neural networks may loosely mimic the manner in which a biological brain works (e.g., via large clusters of biological neurons connected by axons). Each neural unit of a neural network may be connected with many other neural units of the neural network. Such connections can be enforcing or inhibitory in their effect on the activation state of connected neural units. In some embodiments, each individual neural unit may have a summation function which combines the values of all its inputs together. In some embodiments, each connection (or the neural unit itself) may have a threshold function such that the signal must surpass the threshold before it propagates to other neural units. These neural network systems may be self-learning and trained, rather than explicitly programmed, and can perform significantly better in certain areas of problem solving, as compared to traditional computer programs. In some embodiments, neural networks may include multiple layers (e.g., where a signal path traverses from front layers to back layers). In some embodiments, back propagation techniques may be utilized by the neural networks, where forward stimulation is used to reset weights on the “front” neural units. In some embodiments, stimulation and inhibition for neural networks may be more free flowing, with connections interacting in a more chaotic and complex fashion.
In some embodiments, the machine learning model system may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of the predictions. Memory may store training data and one or more trained machine learning models.
As an example, a machine learning model 900 may take inputs 902 and provide outputs 904. In one use case, outputs 904 may be fed back (e.g., active feedback) to machine learning model 900 as input to train machine learning model 900 (e.g., alone or in conjunction with user indications of the accuracy of outputs 904, labels associated with the inputs 902, or with other reference feedback information). In another use case, machine learning model 900 may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its prediction (e.g., outputs 904) and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). In another use case, where machine learning model 900 is a neural network, connection weights may be adjusted to reconcile differences between the neural network's prediction and the reference feedback. In a further use case, one or more neurons (or nodes) of the neural network may require that their respective errors be sent backward through the neural network to them to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model 900 may be trained to generate better predictions.
For example, in some embodiments, inputs 902 may comprise CGM sensor data (e.g., sensor glucose data), and reference feedback information 904 (which feeds back as input to the machine learning model 900) may include clinical data on iCGM performance. For example, the clinical data may be labeled training data (e.g., labeled as outlier, not outlier). Accordingly, when machine learning model 900 receives a particular glucose measurement as input 902, machine learning model 900 may provide an output 904 including a prediction of whether the measurement is an outlier. This outlier information may be used to correct future signals.
In some embodiments, the system may label the clinical data according to known classifications. In some embodiments, the known classifications may be based upon the FDA iCGM special requirements (e.g., the known classifications may be defined such that the outputs 904 conform to the FDA iCGM special requirements). For example, known classifications may include large negative bias, large positive bias, nominal accuracy, or other labels with respect to the iCGM criteria. For example, a large negative bias may indicate that a particular measurement is underestimated, undervalued, or otherwise too low. A large positive bias may indicate that a particular measurement is overestimated, overvalued, or otherwise too high. In some embodiments, these known classifications may describe the CGM sensor data relative to an average measurement or range across the sensor data, an average measurement or range for sensor data associated with a user of the sensor device, or another reference. In some embodiments, the system may take calibration measurements periodically (e.g., every 4-6 hours or any other predetermined period of time) and derive a reference value from the calibration measurements. The system may thus determine positive and negative bias relative to the reference value. Finally, when machine learning model 900 receives a particular glucose measurement as input 902 to, machine learning model 900 may provide an output 904 including a prediction of whether the measurement corresponds to a large negative bias, large positive bias, or nominal accuracy.
In some embodiments, known classifications may include poor accuracy, intermediate accuracy, good accuracy, or other labels. For example, accuracy labels may correspond to wear times, consistency or inconsistency of readings (e.g., as compared to data for a particular user, training data as a whole, etc.), or other indicators of accuracy. In some embodiments, as described above, the system may take calibration measurements periodically (e.g., every 4-6 hours or any other predetermined period of time) and derive a reference value from the calibration measurements. The system may determine accuracy levels relative to the reference value. Accordingly, when machine learning model 900 receives a particular glucose measurement as input 902 to, machine learning model 900 may provide an output 904 including a prediction of whether the measurement corresponds to a poor accuracy, intermediate accuracy, or good accuracy.
In some embodiments, machine learning model 900 may receive a multi-dimensional feature input based on the sensor data. For example, the inputs 902 received by machine learning model 900 may comprise features such as an EIS feature, Isig trending feature, Isig rate of change feature, previous Isig, previous Vcntr, or other feature inputs. Further description of the above feature inputs can be found in U.S. patent Ser. No. 16/773,422, entitled METHODS, SYSTEMS, AND DEVICES FOR CONTINUOUS GLUCOSE MONITORING, which is herein incorporated by reference in its entirety. In some embodiments, the outputs 904 of machine learning model 900 may depend on combinations of the above feature inputs. For example, certain combinations of Isig, EIS, and Vcntr feature inputs may cause machine learning model 900 to classify the signals as outliers (e.g., or large negative bias, large positive bias, poor accuracy, etc.). In some embodiments, other combinations of Isig, EIS, and Vcntr feature inputs may cause machine learning model 900 to classify the signals as not outliers (e.g., or nominal accuracy, intermediate accuracy, good accuracy, etc.). In some embodiments, the machine learning model may use hit criteria (e.g., as described above) to classify the multi-dimensional feature input as an outlier or not an outlier.
In some embodiments, inputs 902 may comprise sensor data from a sensor device and reference feedback information 904 (which is fed back as input to the machine learning model 900) may include clinical data on erroneous sensor use conditions. For example, the clinical data may be labeled training data, which may include a library of error patterns. In some embodiments, the error patterns may be labeled with corresponding erroneous sensor use conditions. For example, erroneous sensor use conditions may include sensitivity loss, loss of signal, signal spiking, environmental errors (e.g., moisture, heat), or other conditions. Accordingly, when a particular glucose measurement is provided as input 902 to machine learning model 900, machine learning model 900 may provide an output 904 including a prediction of an erroneous sensor use condition based on a detected error pattern.
In some embodiments, inputs 902 may further comprise context information relating to the sensor data. In some embodiments, context information may include pattern definition. For example, error patterns may include combinations of patterns of various sensor input features. In this example, context information may include information about the various components (e.g., sensor input features) of the sensor data and how the error pattern affects these components. In some embodiments, the context information may include historic information relating to the sensor data over a time period. For example, such historic information may include information about behaviors, trends, or patterns of the sensor data over a certain time period leading up to a detected error pattern. In some embodiments, behaviors, trends, or patterns of the sensor data may include downward trends, noisy conditions, a history of error patterns, or other context information. In some embodiments, the time period may be several seconds, minutes, hours, or days, or the time period may be the lifetime of the sensor device. The context information may be important for distinguishing between identical error patterns with different underlying erroneous sensor use conditions. For example, for an error pattern (e.g., low signal) within a first context (e.g., stable history), the system may determine that a first erroneous sensor use condition (e.g., temporary signal loss) is associated with the error pattern. In another example, for the same error pattern (e.g., low signal) within a second context (e.g., downward trend), the system may determine that a second erroneous sensor use condition (e.g., sensitivity loss) is associated with the error pattern. The context information is thus vital to determining the correct erroneous sensor use condition associated with a given error pattern and the correct resolution for correcting the erroneous sensor use condition. In this example, output 904 may be further based upon the context information relating to the sensor data.
In some embodiments, inputs 902 may comprise sensor data from a sensor device and reference feedback information 904 (which is fed back as input to the machine learning model 900) may include data on resolutions associated with various erroneous sensor use conditions. For example, the system may use retrospective techniques to apply resolutions to the training data. The system may determine which resolutions are most effective for resolving each erroneous sensor use condition. Thus, the training data may be labeled training data (e.g., error patterns or erroneous sensor use conditions labeled with effective resolutions). Accordingly, when particular sensor data is provided as input 902 to machine learning model 900, machine learning model 900 may provide an output 904 including a prediction of a resolution that is likely to correct for a detected erroneous sensor use condition.
While machine learning model 900 is described in relation to the foregoing examples, it should be understood that the system may train machine learning model 900 to classify or predict characteristics or errors of sensor data according to any other criteria or based on any other inputs. In some embodiments, the system may utilize outputs from machine learning model 900 to determine blanking and termination of signals (e.g., as described below in relation to
In some embodiments, the input data may flow through components 1002, 1004, 1006, and 1008 (i.e., bypassing component 1010). For example, components 1002, 1004, and 1006 may complete sensor feature generation, BG calibration, and SG modelling, respectively, and component 1008 may classify the sensor data and blank or terminate signals accordingly. In some embodiments, conditional blanker and terminator 1008 may include or be associated with machine learning model 900, as shown in
In some embodiments, blanking the CGM sensor data (e.g., via a display interface) may include determining a variable for the CGM calibration algorithm based on the output from machine learning model 900 (e.g., as shown in
In some embodiments, conditional blanker and terminator 1008 may additionally or alternatively terminate a sensor (e.g., stop transmitting any sensor data from that sensor and/or stop displaying any sensor data from that sensor). For example, conditional blanker and terminator 1008 may monitor measurements that have been blanked (i.e., outliers). Conditional blanker and terminator 1008 may track a number of consecutive blanked measurements (e.g., within a certain time frame). For example, conditional blanker and terminator 1008 may use an outlier counter to track blanked measurements and determine when to terminate a sensor. In some embodiments, when machine learning model 900 classifies a measurement as not an outlier, the system may reset the outlier counter (e.g., to zero). In some embodiments, when machine learning model 900 classifies a certain number of measurements in a row as not outliers, the system may reset the outlier counter. For example, machine learning model 900 classifies five measurements as not outliers, the system may reset the outlier counter. In some embodiments, the system may reset the outlier counter based on measurements that machine learning model 900 classifies as having nominal accuracy, intermediate accuracy, good accuracy, etc.
If conditional blanker and terminator 1008 identifies a measurement that has been classified as an outlier (e.g., or large positive bias, large negative bias, poor accuracy, etc.) by machine learning model 900, the system may increase the outlier counter. For example, the system may increment the outlier counter by one. In this example, the outlier counter may continue to increase when outlier measurements are received. As described above, the system may reset the outlier counter at any time in response to receiving one or more measurements classified as non-outliers. The outlier counter may continue to increase in response to outlier measurements until a threshold is breached. For example, the threshold may be predetermined. In some embodiments, the threshold may dynamically change based on wear time, battery life, or other factors. In some embodiments, the threshold may have an associated time limit. For example, in some embodiments, the outlier counter must breach the threshold within a certain time frame (e.g., 1 hour) in order to terminate the signal (e.g., stop transmitting any sensor data from that sensor and/or stop displaying any sensor data from that sensor). In some embodiments, once the system terminates the signal, conditional blanker and terminator 1008 may alert the user that the sensor must be replaced (e.g., via a display interface).
As shown in
Returning to
In some embodiments, error detector and corrector 1010 may identify a resolution in response to a detected erroneous sensor use condition. For example, error detector and corrector 1010 may identify a resolution that is associated with the detected erroneous sensor use condition. In some embodiments, the resolution may be associated with the detected erroneous sensor use condition via a database, library of patterns, library of resolutions, or by some other means. In some embodiments, the resolution may be associated with the detected erroneous sensor use condition via an output from a machine learning model (e.g., machine learning model 900), as discussed above in relation to
In some embodiments, error detector and corrector 1010 may include or be associated with machine learning model 900, as shown in
The above-described embodiments of the present disclosure are presented for purposes of illustration and not of limitation, and the present disclosure is limited only by the claims which follow. Furthermore, it should be noted that the features and limitations described in any one embodiment may be applied to any other embodiment herein, and flowcharts or examples relating to one embodiment may be combined with any other embodiment in a suitable manner, done in different orders, or done in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and/or methods described above may be applied to, or used in accordance with, other systems and/or methods.
The present techniques will be better understood with reference to the following enumerated embodiments:
1. A method comprising: receiving, at a sensor device, CGM sensor data; inputting, at the sensor device, the sensor data in a machine learning model, wherein the machine learning model is trained to identify outlier measurements based on sensor glucose-dependent performance against iCGM criteria using training data comprising clinical data on iCGM performance; receiving, at the sensor device, an output from the machine learning model indicating that the sensor data corresponds to an outlier measurement of the outlier measurements; and blanking the sensor data based on the output.
2. The method of embodiment 1, wherein the machine learning model is trained using a set of training sensor data that is labeled according to known classifications, the known classifications comprising a large negative bias, large positive bias, or nominal accuracy.
3. The method of any of embodiments 1-2, wherein the machine learning model is trained using a set of training sensor data that is labeled according to known classifications, the known classifications comprising poor accuracy, intermediate accuracy, or good accuracy.
4. The method of any of embodiments 1-3, wherein the sensor data comprises a current signal, a voltage signal, or impedance spectroscopy signals.
5. The method of embodiment 4, wherein inputting the sensor data in the machine learning model comprises generating a multi-dimensional feature input based on the sensor data.
6. The method of any of embodiments 1-5, wherein blanking the sensor data based on the output further comprises: determining a variable for a CGM calibration algorithm based on the output; and determining whether to blank the sensor data based on the output from the from the machine learning model.
7. The method of any of embodiments 1-6, wherein the sensor data is received in first time intervals.
8. The method of any of embodiments 1-7, further comprising: resetting an outlier counter based on determining that a first sensor datapoint does not correspond to an outlier measurement; causing the outlier counter to be increased based on determining that a second sensor datapoint corresponds to an outlier measurement; comparing the outlier counter to a threshold; and terminating the sensor device based on determining that the outlier counter has breached the threshold.
9. A method comprising: receiving, at a sensor device, CGM sensor data; inputting, at the sensor device the sensor data into a machine learning model, wherein the machine learning model is trained to identify erroneous sensor use conditions based sensor data error patterns using training data comprising clinical data on erroneous sensor use conditions; and receiving, at the sensor device, an output from the machine learning model indicating an erroneous sensor use condition based on an error pattern identified in the sensor data.
10. The method of embodiment 9, further comprising blanking the sensor data based on the output.
11. The method of any of embodiments 9-10, further comprising flagging a user of the sensor device, on a display interface, based on the output.
12. The method of any of embodiments 9-11, further comprising identifying a resolution associated with the erroneous sensor use condition identified in the sensor data.
13. The method of embodiment 12, further comprising implementing the resolution by manipulating the sensor data, on a display interface, of the sensor device.
14. The method of embodiment 12, further comprising implementing the resolution by manipulating the sensor data received from the sensor device.
15. The method of any of embodiments 9-14, further comprising: receiving input indicating context information relating to the sensor data; inputting the context information relating to the sensor data into the machine learning model; and wherein the output from the machine learning model indicating an erroneous sensor use condition is further based on the context information relating to the sensor data.
16. The method of embodiment 15, wherein the context information relating to the sensor data includes historic information relating to the sensor data over a time period.
17. The method of claim 9, further comprising training the machine learning model to identify the erroneous sensor use conditions based on the sensor data error patterns using the training data comprising the clinical data on the erroneous sensor use conditions.
18. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising those of any of embodiments 1-17.
19. A system comprising: one or more processors; and memory storing instructions that, when executed by the processors, cause the processors to effectuate operations comprising those of any of embodiments 1-17.
20. A system comprising means for performing any of embodiments 1-17.
Claims
1. A sensor device for applying machine learning models to improve integrated continuous glucose monitoring (“iCGM”) performance of continuous glucose monitoring (“CGM”) calibration algorithms, the sensor device comprising:
- memory configured to store a machine learning model, wherein the machine learning model is trained to identify outlier measurements from CGM sensor data based on sensor glucose-dependent performance against iCGM criteria using training data comprising clinical data on iCGM performance; and
- a processor configured to: receive CGM sensor data; input the sensor data in the machine learning model; receive an output from the machine learning model indicating that the sensor data corresponds to an outlier measurement of the outlier measurements; and blank the sensor data based on the output.
2. The sensor device of claim 1, wherein the machine learning model is trained using a set of training sensor data that is labeled according to known classifications, the known classifications comprising a large negative bias, large positive bias, or nominal accuracy.
3. The sensor device of claim 1, wherein the machine learning model is trained using a set of training sensor data that is labeled according to known classifications, the known classifications comprising poor accuracy, intermediate accuracy, or good accuracy.
4. The sensor device of claim 1, wherein the sensor data comprises a current signal, a voltage signal, or impedance spectroscopy signals.
5. The sensor device of claim 4, wherein, to input the sensor data in the machine learning model, the processor is further configured to generate a multi-dimensional feature input based on the sensor data.
6. The sensor device of claim 1, wherein, to blank the sensor data based on the output, the processor is further configured to:
- determine a variable for a CGM calibration algorithm based on the output; and
- determine whether to blank the sensor data based on the output from the machine learning model.
7. The sensor device of claim 1, wherein the sensor data is received in first time intervals.
8. The sensor device of claim 1, wherein the processor is further configured to:
- reset an outlier counter based on determining that a first sensor datapoint does not correspond to an outlier measurement;
- cause the outlier counter to be increased based on determining that a second sensor datapoint corresponds to an outlier measurement;
- compare the outlier counter to a threshold; and
- terminate the sensor device based on determining that the outlier counter has breached the threshold.
9. A method of applying machine learning models to improve integrated continuous glucose monitoring (“iCGM”) performance of continuous glucose monitoring (“CGM”) calibration algorithms by classifying outlier measurements based on iCGM criteria, the method comprising:
- receiving, at a sensor device, CGM sensor data;
- inputting, at the sensor device, the sensor data in a machine learning model, wherein the machine learning model is trained to identify outlier measurements based on sensor glucose-dependent performance against iCGM criteria using training data comprising clinical data on iCGM performance;
- receiving, at the sensor device, an output from the machine learning model indicating that the sensor data corresponds to an outlier measurement of the outlier measurements; and
- blanking the sensor data based on the output.
10. The method of claim 9, wherein the machine learning model is trained using a set of training sensor data that is labeled according to known classifications, the known classifications comprising a large negative bias, large positive bias, or nominal accuracy.
11. The method of claim 9, wherein the machine learning model is trained using a set of training sensor data that is labeled according to known classifications, the known classifications comprising poor accuracy, intermediate accuracy, or good accuracy.
12. The method of claim 9, wherein the sensor data comprises a current signal, a voltage signal, or impedance spectroscopy signals.
13. The method of claim 12, wherein inputting the sensor data in the machine learning model comprises generating a multi-dimensional feature input based on the sensor data.
14. The method of claim 9, wherein blanking the sensor data based on the output further comprises:
- determining a variable for a CGM calibration algorithm based on the output; and
- determining whether to blank the sensor data based on the output from the machine learning model.
15. The method of claim 9, wherein the sensor data is received in first time intervals.
16. The method of claim 9, further comprising:
- resetting an outlier counter based on determining that a first sensor datapoint does not correspond to an outlier measurement;
- causing the outlier counter to be increased based on determining that a second sensor datapoint corresponds to an outlier measurement;
- comparing the outlier counter to a threshold; and
- terminating the sensor device based on determining that the outlier counter has breached the threshold.
17. A non-transitory computer-readable media for continuous glucose monitoring comprising instructions that, when executed by one or more processors, cause operations comprising:
- receiving, at a sensor device, CGM sensor data;
- inputting, at the sensor device, the sensor data in a machine learning model, wherein the machine learning model is trained to identify outlier measurements based on sensor glucose-dependent performance against iCGM criteria using training data comprising clinical data on iCGM performance;
- receiving, at the sensor device an output from the machine learning model indicating that the sensor data corresponds to an outlier measurement; and
- blanking the sensor data based on the output.
18. The non-transitory computer-readable media of claim 17, wherein the instructions cause further operations comprising:
- determining whether a set of training sensor data in the training data corresponds to known classification, wherein the known classification comprises a large negative bias, large positive bias, or nominal accuracy;
- labeling the set of training sensor data with the known classification; and
- training the machine learning model using the labeled set of training sensor data.
19. The non-transitory computer-readable media of claim 17, wherein the instructions cause further operations comprising:
- determining whether a set of training sensor data in the training data corresponds to known classification, wherein the known classification comprises poor accuracy, intermediate accuracy, or good accuracy;
- labeling the set of training sensor data with the known classification; and
- training the machine learning model using the labeled set of training sensor data.
20. The non-transitory computer-readable media of claim 17, wherein the instructions cause further operations comprising:
- resetting an outlier counter based on determining that a first sensor datapoint does not correspond to an outlier measurement;
- causing the outlier counter to be increased based on determining that a second sensor datapoint corresponds to an outlier measurement;
- comparing the outlier counter to a threshold; and
- terminating the sensor device based on determining that the outlier counter has breached the threshold.
Type: Application
Filed: Dec 14, 2020
Publication Date: Jun 16, 2022
Inventors: ELAINE GEE (Windsor, CA), JEFFREY NISHIDA (Chicago, IL), PETER AJEMBA (Canyon Country, CA), KEITH NOGUEIRA (Mission Hills, CA), ANDREA VARSAVSKY (Santa Monica, CA)
Application Number: 17/121,624