SYSTEM AND METHODS FOR DETERMINING HEALTH-RELATED METRICS FROM DATA COLLECTED BY A MOBILE DEVICE
Techniques and systems include predicting various health conditions using a photoplethysmography (PPG) signal or a video signal based on images of a patient's fingertip or other tissue or other body portion captured using a mobile device, such as a smartphone or tablet. The video signal may be transformed into a pseudo PPG signal to measure blood volume changes in the patient's blood flow to derive data indicating a disease state or health-related characteristic, such as blood oxygen level, blood glucose level, heart rate variability, hemoglobin, respiration rate, or arrhythmia. Techniques involve real-time environment assessment and problematic issue detection, training an artificial intelligence (AI) model to measure signal quality so as to select high-quality signals from a range of signals, and domain adaption and transfer learning to make use of publicly available datasets.
This application claims priority and benefit from the U.S. Provisional Patent Application 63/226,541, filed Jul. 28, 2021, and titled, “System and Methods for Determining Health-Related Metrics from Data Collected by a Mobile Device,” which are incorporated herein by reference in their entirety for all purposes.
BACKGROUNDThere are several quantities that are regularly measured and evaluated to provide information about a person's health. These quantities and both their real-time values and trends over various timescales can sometimes be used as indications of disease or a serious health condition. Examples of such quantities that may be used to monitor and, in some cases, diagnose a health condition include blood pressure, pulse or heart rate, and blood oxygen saturation (SpO2), among other biomarkers. Conventionally, such health-related quantities are measured at a health clinic, the office of a provider of medical services, or with special equipment that is used in the home.
However, these conventional methods of measuring health quantities are not always available, may require advance planning to use, and are not convenient for situations where a person is engaging in activities away from home. Further, the conventional methods provide a “snapshot” in time of the values of the health-related quantities but may not provide sufficient data or data collected at a sufficient time or set of times to detect or diagnose a health condition. For example, a physician or paramedic may want to know a person's blood pressure over a time interval around when they suffered some type of pain. This may be difficult to do without the person being in a location where someone can make those measurements. As another example, a person may want to monitor how their blood pressure responds to certain activities or behaviors, and such monitoring may be difficult to do outside of their home.
Although there are some portable devices that can be used to monitor blood pressure, these typically are not as accurate as desired. Further, although portable blood pressure monitors are available, these typically require that a person be in a situation where a “cuff” may be used to obtain the blood pressure reading, and this is not always possible or convenient.
What is desired are systems, apparatuses, and methods for enabling a person to monitor one or more of their health-related quantities more conveniently using a ubiquitous mobile device and without having to be at a particular location.
The disclosure will be understood more fully from the detailed description given below and from the accompanying figures of embodiments of the disclosure. The figures are used to provide knowledge and understanding of embodiments of the disclosure and do not limit the scope of the disclosure to these specific embodiments.
Photoplethysmography (PPG) is an optical technique used to detect volumetric changes in blood in peripheral circulation, using low-intensity infrared (IR) light. When light travels through biological tissues it is absorbed by bones, skin pigments, and both venous and arterial blood. Since light is more strongly absorbed by blood than surrounding tissues, changes in blood flow can be detected by PPG sensors as indicated by changes in the transmitted intensity of light. A voltage signal from PPG sensors may be proportional to the quantity of blood flowing through blood vessels. Even small changes in blood volume can be detected using this method. A PPG signal has several components including volumetric changes in arterial blood that is associated with cardiac activity, variations in venous blood volume that modulates the PPG signal, and a DC component showing the tissues' optical properties and subtle energy changes in the body.
This disclosure describes a number of techniques and systems for determining or predicting various health conditions. The techniques and systems involve use of an optical signal, such as a PPG signal or a video signal transformed into a pseudo PPG signal, to measure blood volume changes in a patient's blood flow to derive data indicating a disease state or biomarker. In the past, an optical signal used for this purpose generally suffered from quality and inconsistency issues, even in a controlled environment on the same patient.
Previous efforts to derive blood pressure from smartphone-sourced PPG have shown promising results in lab settings but have not reached good enough accuracy in real-world settings for widespread adoption. A substantial technical barrier to achieving sufficient real-world accuracy is the handling of the inconsistency and quality of these PPG signals created under a diversity of conditions set forth by sensors (e.g., camera hardware, numerous flash settings), users (e.g., body weight, skin tone, ages), and environments (e.g., contact grips, lighting conditions). There is no canonical “ground truth” PPG, since even purpose-built PPG devices in controlled environments may produce different signals for the same subject by small changes in placement or orientation of the device.
Machine learning (ML) has been used to attempt to address the issues and complexity described above. Traditional ML methods such as a Support Vector Machine (SVM) algorithm or regression have shown promising results but require perhaps more than fifty or so features extracted by hand, even when using PPG signals from purpose-built devices in controlled environments. In recent years, neural networks have been used in different ways to successfully improve performance, especially with smartphone-sourced PPG. But what has been achieved in this way is still insufficient due to the large diversity of signals and expensive, limited training data. One common theme is that poor signals are often thrown out during training and are not considered during inference, leading to many failed or inaccurate readings. Another common theme is that models may work only on a limited number of smartphones or other devices.
Embodiments described herein address the problems discussed above and may achieve real-world accuracy with even just limited data. The embodiments involve real-time environment assessment and problematic issue detection, training an AI model to measure signal quality so as to select high-quality signals among signals having a range of qualities, and domain adaption and transfer learning to make use of publicly available datasets, just to name a few examples.
In particular, embodiments described herein are directed to systems, methods, and apparatuses for more conveniently and accurately measuring and monitoring one or more health-related characteristics, including but not limited to blood pressure and heart rate, for example. In some embodiments, the health-related characteristics may include blood oxygen level, blood glucose level, hemoglobin A1C, heart rate variability, hemoglobin, respiration rate, blood oxygen, or arrhythmia.
One embodiment is directed to a method for determining one or more health-related quantities for a person based on video images captured using a mobile device (e.g., a smartphone). The method may include steps, stages, or operations that include activation or launch of an application hosted by the mobile device and collecting a sample or samples of video of a person's fingertip or tissue from another part of the body. For example, during data collection, participants may have equipment placed on them, such as a pulse oximeter placed on the finger, toe, and/or earlobe, a single-lead Electrocardiogram (ECG), or an Inertial Measurement Unit (IMU) taped to the sternum. Collection of data from these sensors may include a participant's blood pressure, which may be taken using the auscultation method (e.g., using a stethoscope to listen for Korotkoff sounds).
The method may also include processing the collected sample(s) to approximate a PPG signal and detection of possible data collection problems during acquisition of user data, which can occur under normal use in diverse conditions. The method may further include evaluating quality of the PPG signal that may have been gathered from diverse subjects and cameras under diverse conditions and passing the PPG signal as input to a trained blood pressure AI model.
The activation or launch of the application may involve the application setting collection parameters and equipment characteristics, such as device manufacturer and model, camera or video settings, lens, or video processing options, and so on. If needed, in some embodiments, the application may collect data regarding the patient, such as skin tone, and other data regarding local environment, lighting conditions, stability/motion of device, and so on, before beginning video capture. Collecting a sample or samples of video from a person's fingertip (or other tissue) using a mobile device may involve use of the application's user interface (UI) to guide a user in the placement of their fingertip and in holding the mobile device correctly and steadily to collect the data. A camera (and flash LED) of a mobile device, such as a smartphone, or other portable device may be used.
Processing the collected sample(s) to approximate a PPG signal may involve a PPG signal generator module or similar functionality that may continuously process raw camera frames by extracting average red (R), green (G), and blue (B) values of all the RGB pixels in each video frame. In some implementations, the red channel is taken to approximate the PPG signal. Obtained red channel signals may be smoothed using a Butterworth filter or other filter and up-sampling may be performed using linear interpolation, for example. A Butterworth filter is a type of signal processing filter designed to have a frequency response as flat as possible in the passband. It may also be referred to as a maximally flat magnitude filter.
In some implementations, data collection problems (problematic issues) during acquisition of user data may be detected. Such problems may occur under normal use in diverse conditions. In other words, issues or factors that could result in unreliable or unusable data may be detected. For example, ambient conditions or factors may include motion of camera (e.g., relative to finger), movement of finger (e.g., relative to camera), improper position of finger, and a change in lighting that impacts an ability to accurately and reliably extract a PPG-like signal from camera frames. An Issue Detector module or such functionality may be included in the system architecture, as described below.
Camera settings may be adjusted to get a “good” (e.g., acceptable) signal during diverse conditions (ambient lighting, skin tone of user, mobile device camera, etc.). In some implementations, previously measured or entered skin tone information of a user may be retrieved from memory of the user's mobile device. An output that includes a determined disease state or a biomarker may be modified based on the retrieved skin tone information. In some embodiments, to at least partially control ambient factors/conditions, an Environment Assessor module may step up the light sensitivity periodically (e.g., every 1.2 seconds), subsequently holding constant the light sensitivity when the red channel intensity, for example, from the camera frames reach a minimum (predetermined) threshold. The Environment Assessor module may also adjust the intensity of light emitted by a flash LED system of the camera.
Evaluating the quality of the PPG signal gathered from diverse subjects and cameras under diverse conditions may involve identifying individual pulses in the PPG signal. For example, in one embodiment, a Pulse Generation module or such functionality may identify a peak or valley that expresses an individual pulse in the PPG signal. A signal quality evaluation process may reject or remove from further processing pulses that fail to meet the threshold required for processing and evaluation. In one embodiment, a Signal Quality Assessor module may be a neural network trained to assess individual pulses. During training of the Signal Quality Assessor module, collected fingertip data (or similar type of data for other tissue or body part) may be manually annotated (described below) with an “acceptable” or “not acceptable” label to produce training data for a model. For example, an “acceptable” pulse is uniform (multiple pulses are consistent among themselves) and has the characteristics of a PPG signal (e.g., generally having a steep slope up to a relatively sharp peak and having zero or up to 2 smaller notches on the way down).
Based on environmental and signal quality conditions (during signal acquisition), and an absence of data acquisition problems, the PPG pulses may be passed as input to a trained AI blood pressure model. In one embodiment, the blood pressure model is a neural network that continuously takes in “acceptable”-quality pulses from the PPG signal as input and produces systolic blood pressure (BP) and diastolic BP as output. The model averages the produced systolic and diastolic BP values to produce final BP values that are returned to the user.
As mentioned above, techniques that may achieve real-world accuracy may involve training an AI model to measure signal quality so as to select high-quality signals from signals having a range of qualities. In one embodiment, an AI model training process may involve, due to limited training data, a base model that is trained on publicly available, synchronized, PPG and BP datasets. The trained base model may be used as a basis for continued training using learning on collected (e.g., historical) pulse oximeter PPG, smartphone fingertip video, and blood pressure data, as described below. Although examples described herein involve a fingertip, other tissue or parts of the body may be used instead of, or in addition to, the fingertip. Claimed subject matter is not limited in this respect. A smartphone-sourced PPG signal produced from fingertip videos may also be used by applying a Butterworth filter, and up-sampling the pulse oximeter PPG and smartphone-sourced PPG to match the frequency of the publicly available data using linear interpolation, for example.
Additional AI training data may be captured for use in training the base model, such as from users of the application, training data collected under controlled conditions, and continuous data acquisition and model training through data collection along with accurate measurement of health quantity using a separate device(s). When in use, data collection and processing may be performed on a user's device, wherein the user may maintain privacy.
Output from the trained AI model may provide information representing a measure of one or more of the person's blood pressure, heart rate, SpO2, or other health-related quantity. Based on the model output, one or more tasks may be executed by an application hosted by a mobile device, such as providing results via an application programming interface (API) call back that enables a developer to use the data in their own application, showing trends in the measured quantities over time, tuning medication dosages based on changes in the measured quantities, placing the measured data into an electronic health record (EHR) for use by a physician, and statistically analyzing the measured data in conjunction with other data or information (such as sleep quality, stress, or anxiety levels, etc.). Other tasks that may be performed by a mobile device, for example, may include generating an alert to a user that a measured quantity exceeds a preset threshold value, wherein the average value of a measured quantity exceeds the preset value over a set time period, and wherein the measured value in combination with another aspect of the user (BMI, weight, blood glucose level, etc.) suggests a problem needing attention. Still other tasks may include generating a phone call to an emergency number if a measured quantity alone or over time exhibits a characteristic of serious medical problem (such as stroke, etc.) and generating a recommendation to engage in an activity, undertake a change in behavior, etc.
As described above, in various embodiments, biomarkers may be estimated from biosensor data, which may be an optical signal like a PPG, electrical signals like an ECG, and so on. The optical biosensor data may be acquired from a smartphone-captured video of a person's finger or from a PPG sensor in devices such as a pulse oximeter. For example, measurements of real-time blood pressure and heart rate data may be acquired via fingertip video obtained with a mobile device. In an implementation, the mobile device may include a data measurement module (DMM), which may comprise a software development kit (SDK) embedded into an operating system of the mobile device. A user may use the mobile device (e.g., a smartphone, wrist-worn wearable such as smartwatch or smart wristband, earbud, or tablet) to measure their blood pressure and heart rate, for example, by operating the DMM, which may navigate the user to a particular view (e.g., menu of options and/or instructions) on the smartphone. The user may place their fingertip against the lens of the mobile device camera, and after about a minute or so, for instance, the DMM may provide the user's blood pressure and heart rate data to an application on the mobile device or external to the mobile device. In some implementations, video frames and related data may be processed completely on the device so that user identities may be completely unknown outside the device, for anonymity.
In some embodiments, the DMM works by extracting the red channel from each frame of the fingertip video to form a PPG signal. As explained above, PPG is commonly used in methods for measuring blood volume changes and can be used to derive health measures like heart rate, Sp02, and blood pressure. A neural network may be used to extract these health measures from the smartphone signal produced by the DMM.
Conditions that exist during image capture may be important and thus considered during image or video processing, as explained below. Lighting conditions, motion conditions, and orientation of the smartphone may be important by affecting the quality of the images or the video that will subsequently be used to arrive at a pseudo PPG signal. Smartphone 108 may include an onboard flash LED 110, an onboard light sensor 112, a capacitive touch sensor 113, and an onboard accelerometer 114 (or IMU). Flash LED 110 may provide light to the fingertip and the surrounding area. This may light illuminate fingertip 104 with an intensity that allows for successful image capture by camera 106. The biomarker application hosted by the smartphone may at least partially control the intensity of light emitted by flash LED 110. Light sensor 112 may measure the intensity of ambient light 116 impinging on the light sensor. This intensity may be indicative of illumination of fingertip 104 and the surrounding area. Light sensor 112 may provide its measurements to the biomarker application hosted by the smartphone. In turn, the application may control the flash or exposure settings of the camera based on the light sensor measurements.
Capacitive touch sensor 113 may produce a signal based on whether a fingertip or other tissue is in contact with the camera (or a lens or lens cover thereof) and accelerometer 114 may measure motion and orientation of smartphone 108. Such motion may be vibratory motion or longer time-constant motion, for example. For successful image capture that can lead to high-quality pseudo PPG signals, relative motion between fingertip 104 and camera 106 should be small or non-existent. As described below, measurements by light sensor 112, capacitive touch sensor, and accelerometer 114 may be used by the biomarker application to assess the quality of captured images, video, or resulting pseudo PPG waveforms. For example, such measurements may be grouped as metadata for the captured images or video. Accordingly, the metadata can be used to determine the conditions that existed during acquisition of the images or video.
Client application 302 may comprise computer-executable instructions that are installed on a user's device, which may be a mobile device such as a smartphone, wrist-worn wearable such as smartwatch or smart wristband, earbud, or tablet. In some embodiments, the application uses APIs to integrate functionality for measuring blood pressure or other health-related quantities with the capabilities of the mobile device. Client application 302 may use the APIs to navigate to a user interface associated with a video/data collection service and to provide blood pressure values when complete. In some embodiments, client application 302 may be developed using an SDK that is provided to enable access to, and use of, models and data processing methods, described below.
Biomarker Detector UI 304 is a user interface component that guides (e.g., provides real-time instructions for) the user in measuring their blood pressure or other biomarkers. Biomarker Detector UI 304 (and underlying services or functions) may provide tools to enable the user to initiate the video/data collection used to determine their blood pressure and/or provide a mechanism to interact with other features or functionality of system 300 and methods described herein. For example, the Biomarker Detector UI may display a view that provides instructions and shows feedback to the user, captures camera frames, and continuously or periodically passes the frames to PPG Signal Generator 306. For example, Biomarker Detector UI 304 may guide the user in placing their fingertip correctly on the camera, holding the smartphone correctly and steadily, and may provide the user with measurement progress and may indicate any errors in the process. Client application 302 may determine such measurement progress based, at least in part, on measurements received from accelerometer 114, for example. Biomarker Detector UI 304, via client application 302, may keep the camera flash LED on, and may control its intensity, to illuminate the finger during image capture.
Biomarker Detector UI 304 may be part of a service or application that interacts with Issue Detector 308 to control the acquisition of video/data and processing of the acquired video/data into a PPG signal by PPG Signal Generator 306. In some embodiments, Biomarker Detector UI 304 may be provided as a part of an SDK. In some embodiments, Biomarker Detector UI 304 may allow for measurements of a different health-related quantity (e.g., other than, or in addition to, blood pressure). PPG Signal Generator 306 may provide PPG signals to Issue Detector 308 and Biomarker Detection module 310.
A blood pressure detection process, performed by Biomarker Detection module 310, may begin once no issues (e.g., as determined by Issue Detector 308) have been identified during an initial time period, such as the first five seconds of the process, for example. A PPG signal may be passed through the following components to produce blood pressure: an Environment Assessor 312, a Signal Quality Assessor 314, and a Biomarker model 316. There generally may be many variables that impact the quality of the PPG signal that in turn impact the accuracy of Biomarker readings. Such variables may include ambient lighting conditions, skin tone, and smartphone camera quality. To control for this, Environment Assessor 312, via client application 302, may hold constant the light sensitivity and shutter speed of the camera (e.g., 106) of the mobile device at values that produce a minimum threshold of intensity. In some implementations, Environment Assessor 312 may step up the light sensitivity periodically, such as every 10 seconds, for example. In another implementation, Environment Assessor 312 may step up the light sensitivity by 75 ISO every 1.2 seconds, locking the light sensitivity when the red channel intensity from the camera frames reaches a minimum threshold. These values (e.g., 75 ISO and 1.2 s) may be determined empirically to increase the red channel intensity as fast as possible while stabilizing at each increment. Claimed subject matter is not limited to such values.
An updated PPG signal with camera settings (e.g., as metadata, such as time stamp of image capture, exposure time, aperture, resolution, and so on) are passed through Signal Quality Assessor 314, which may produce individual pulses from signals passed from Environment Assessor 312 and may pass acceptable pulses to Biomarker model 316. In one embodiment, Signal Quality Assessor 314 is a neural network trained to assess individual pulses. Training data may be manually annotated based on collected user fingertip data with a label of “acceptable” or “not acceptable” to produce training data for this model. In general, an “acceptable” pulse is uniform (consistent with other pulses) and has the characteristics of a PPG signal (generally a steep slope up to a relatively sharp peak and has 0 to 3 smaller notches on the way down), though claimed subject matter is not so limited.
Biomarker model 316, which is a neural network, may depend on relatively high quality signals to produce a good result. In some implementations, a time-series of image frames of video are partitioned into video chunks (e.g., segments) that each have a predetermined time span. Accordingly, Signal Quality Assessor 314 looks at such chunks of signals provided by Environment Assessor 312 and only passes acceptable signals to Biomarker model 316. For example, such chunks of signals may be two-second long segments of a PPG signal. In some implementations, two contiguous video chunks may be combined into a video segment having a portion that comprises an overlap between the two contiguous video chunks. Signal Quality Assessor 314 may generate a quality score determined by comparing a signal quality feature of a PPG signal provided to Biomarker Detection 310 to a same-type signal quality feature of a deep learning-based model PPG signal, which may be stored in computer memory or updated and provided by Biomarker Model 316, for example. In other implementations, Signal Quality Assessor 314 may generate a quality score based on metadata included in the PPG signal provided to Biomarker Detection 310.
In some implementations, Signal Quality Assessor 314 is a neural network trained to assess chunks of PPG signals. Biomarker Model 316 may produce a systolic BP and diastolic BP as output. The model may average the produced systolic and diastolic BP values to produce final BP values that are returned to the user. After acceptable PPG/red channel signals have been processed, the blood pressure values may be provided to the user through client application 302. In some embodiments, the blood pressure values, peak values, changes over time, and so on may be used as input data to a set of rules or a trained model to determine whether to generate an alert, place a call to a medical provider, recommend an activity or change in behavior to the user, etc. In some embodiments, Biomarker model 316 is based, at least in part, on a neural network that is trained using transfer learning on pulse oximeter PPG data and information collected or produced by a mobile device hosting client application 302, as described below. Such information may include video chunks or blood pressure data previously determined by the mobile device, for example.
In some embodiments, as mentioned above, fingertip data, such as in the form of PPG segments, may be annotated (automatically or manually) as being “acceptable” or “not acceptable” to produce training data for Biomarker model 316, which is a neural network that continuously or periodically takes in as input, “acceptable”-quality, segments (e.g., two-second long chunks) of the PPG signal and produces systolic BP and diastolic BP as output. Biomarker model 316 may average the produced systolic and diastolic BP values to produce final BP values that are returned to the user. In some implementations, the training process for the Biomarker model 316 may be performed by first training a base model on publicly available, synchronized, PPG and BP datasets. The trained base model may continue to be trained using transfer learning on pulse oximeter PPG, smartphone fingertip video, and blood pressure data subsequently collected. The smartphone PPG may be produced from fingertip videos in a similar process on the mobile device. For example, this process may include using red channel data of a PPG, applying a Butterworth filter, and up-sampling pulse oximeter PPG and smartphone PPG to match the frequency of the publicly available data using linear interpolation.
PPG Signal Generator 306 may receive video frames collected by a camera in the user's device and may perform the signal/image processing to generate the PPG/red channel signal for further analysis and evaluation. For example, PPG Signal Generator 306 may continuously or periodically process raw image frames by extracting the average red, green, and blue pixel values across an array of pixels in each frame. In some implementations, a single image frame at a time is processed. In other implementations, two or more image frames are processed at a time. The red values may be smoothed with a Butterworth filter and up-sampled via linear interpolation for Biomarker model 316. Such processed frames and associated PPG signal may be passed to the UI of client application 302 as “detection complete,” as indicated by arrow 318.
Issue Detector 308 may be responsible for implementing a set of processes to determine if the video/data collection should be interrupted, stopped, or discarded due to unreliability caused by one or more of the user's finger position, motion of the camera or user's finger, or ambient light conditions, among other possible sources of error. Patient physiological data, in the form of a PPG signal, for example, may include metadata, which may be measured by an IMU of the camera, or other elements that may be used as signal quality features. The patient physiological data, which may also include a disease state or a biomarker feature, may be provided to Issue Detector 308 from PPG Signal Generator 306, for example. Signal quality features and disease state or biomarker features may be manifested as characteristics or parameters of the PPG signal, for example. For example, a PPG signal having such characteristics and parameters may be provided to Issue Detector 308 to determine signal quality (e.g., a quality score) or whether problematic issues occurred during the measurement process that led to the PPG signal. The PPG signal having such characteristics and parameters may also be provided to Biomarker Detection 310 to determine a disease state or biomarker. For example, a process for determining the disease state or biomarker of patient physiological data based on a disease state or a biomarker feature and a quality score may include determining that the quality score exceeds a threshold acceptance value and determining the disease state or the biomarker of the patient physiological data based on the disease state or biomarker feature.
Issue Detector 308 may use such metadata to determine ambient or environmental conditions that existed during the measurement process that produced the patient physiological data. For example, metadata measured by an IMU (e.g., or an accelerometer) of the camera may include relative motion between the camera and a user's fingertip and the orientation of the camera (e.g., and thus the orientation of the mobile device housing the camera).
In some embodiments, translational and rotational motion of the camera may be represented by one or more motion signatures, which may be represented by a waveform or plot of values. For example, translational motion may be a plot or table of time-dependent values of displacement of the camera relative to a fixed point in space. The plot or table may lead to a motion signature for the camera during a time span. Similarly, rotational motion may be one or more plots or tables of time-dependent values of pitch, yaw, and roll of the camera relative to a fixed axis. The plots or tables may lead to other types of motion signatures for the camera during a time span. Various features of the motion signatures may be analyzed. Such features may include frequency-of-motion (e.g., Fourier analysis of motion signatures in the frequency domain), shapes of waveforms or curves, and duration of such features (e.g., when they occurred relative to the time of data acquisition and for how long they occurred), just to name a few possible features. For example, a motion signature expressing displacement of the camera may include a feature that indicates that the camera was not resting on a stable surface (e.g., a tabletop). Such a feature may be a relatively low frequency (e.g., 5-10 Hz) component of the motion signature, which may be a result of the camera being hand-held during data acquisition. This type of motion may likely adversely affect the quality of data acquired during the time of this motion because the proximity of the fingertip to the camera may be difficult to hold constant with this type of motion. In another example, a motion signature expressing angular displacement, such as a relatively slow roll, of the camera may include a feature that also indicates that the camera was not resting on a stable surface. This type of motion, however, may not affect the quality of data acquired during the time of this motion because the proximity of the fingertip to the camera may nevertheless be held constant with this type of motion. Such dynamics of camera motion, as given in these and other examples, may be captured in motion signatures. Accordingly, analyzing these dynamics and how they affect motion of a fingertip or other tissue during data acquisition may allow for assessing the quality of the data.
In other embodiments, relative motion between the fingertip and the camera may be determined based, at least in part, on intensity measurements of pixel data of multiple image frames. For example, substantial changes of light intensity among image frames may indicate to Issue Detector 308 that relative motion occurred during capture of these image frames. In still other embodiments, relative motion between the fingertip and the camera may be determined based, at least in part, on motion signatures in combination with intensity measurements of pixel data of multiple image frames. In still other embodiments, relative motion between the fingertip and the camera may be determined to be zero if a signal from a capacitive touch sensor (e.g., 113), which may be onboard the handheld device and relatively near or on the lens of the camera, indicates continuous contact between the fingertip and the camera during data acquisition.
If an issue is detected, then Issue Detector 308 may send a signal or message to Biomarker Detector UI 304, as indicated by arrow 320, to discard or prevent the further processing of particular samples of the collected frames of video/data. Moreover, Issue Detector 308 may send a signal or message to Biomarker Detector UI 304 including instructions for the Biomarker Detector UI to adjust in real-time a parameter of ambient conditions, the adjusting being based on the determination, by the Issue Detector, that the signal characteristic of the pseudo PPG signal does not meet a quality criterion. The ambient conditions may include relative motion between the camera and the user's finger and the ambient light to which the sensor is exposed. Such adjustments in real-time may occur during the measurement process.
In particular, Issue Detector 308 may continuously or periodically look for issues that prevent production of high-quality signals. System 300 may include a detector, or process of detection, for each problematic issue, which, upon detection, may be provided as feedback to the user, via the UI of client application 302. For example, if a detector detects a problematic issue, the user may be alerted and Biomarker Detection module 310 may be reset.
As mentioned above, some examples of problematic issues may include a finger-off condition, motion/unstable ambient lighting condition, and phone orientation problem, just to name a few examples. In a particular implementation, a finger-off condition may be detected by gathering the last 500 milliseconds (ms) of video frames and analyzing the ratios of red to blue and green to identify whether the user's finger is correctly on the camera. A threshold for the ratio values may be dynamically selected based on the overall intensity in the collected frames to account for variables such as skin tone and camera settings. These thresholds may be identified empirically through collected fingertip video data where fingers are determined to be on or off at different intensities and camera settings. The image frame may also be segmented into quadrants (e.g., or other proportions), with ratios and thresholds being determined for each quadrant, with finger-off condition being identified if at least one quadrant's ratio is below a threshold. In another implementation, a finger-off condition may be detected by analyzing motion signatures of the camera, as described above.
In a particular implementation, a motion/unstable ambient lighting condition may be detected by gathering the last 500 ms of video frames and looking for the difference between minimum and maximum red pixel values. If the difference is more than a threshold value (e.g., 20 in a range of 0-255), it may be determined that too much motion has occurred or that the ambient lighting is too unstable. A phone orientation problem may occur if the user does not hold the phone relatively flat, thus affecting the amount of ambient light that reaches the camera. An on-board accelerator in the phone may be used to determine if the phone is flat, for example.
Assuming that no problematic issues, such as those described above, are detected, PPG Signal Generator 306 may generate a PPG signal/red channel, as indicated by arrow 322, for further analysis and evaluation by Biomarker Detection module 310.
In some embodiments, the services and functionality of system 300 may be combined with different ground truth data to generate measures of other health-related quantities and/or to provide greater insight into a user's health. For example, system 300 may be used to provide one or more of the following measurements if the appropriate ground truth is available: Sp02, involving use of a hypoxia tent to lower the subject's blood oxygen level and use of the Sp02 readings from a pulse oximeter; Blood glucose, involving the taking of blood glucose measurements via a finger stick and capturing data from a user before and after a meal; Hemoglobin A1C (HbA1c), involving taking a HbA1c measurement via a finger stick and lab analysis or point-of-care HbA1c analyzer; Heart rate variability, using an ECG device; Hemoglobin, using a hemoglobin measurement device; Respiration rate, by attaching a respiration rate band onto the user; and Arrhythmia, using an ECG device. Issue Detector 308, Environment Assessor 312, and Signal Quality Assessor 314 described above as applied to blood pressure and heart rate measurements may instead or also be used to make measurements of Sp02, blood glucose, heart rate variability, hemoglobin, respiration rate, and arrhythmia. Further, in some embodiments, the camera (image/video sensor) of the mobile device may be combined with other sensors on the device to collect additional data and signals that may be used to assist in evaluating a user's health or condition.
As discussed above, biomarkers, such as blood pressure, hbA1c, hypertension risk, and heart rate, for example, may be estimated based on biosensor data, such as a PPG. Optical biosensor data may be acquired from a smartphone-captured video of a person's finger using a system such as 300. For example, measurements of real-time blood pressure and heart rate data, or other biomarker, may be acquired via fingertip video obtained with a mobile device. In this context, for example, “real-time” refers to a situation where such biomarkers may be provided to a user while the user continues to be measured for the biomarkers, the latter measurements perhaps providing updated measurements. In an implementation, a user may use a smartphone, including a DMM, to measure their blood pressure and heart rate by placing their fingertip against the lens of the smartphone camera.
In some embodiments, a PPG signal may be acquired from a smartphone by averaging all pixels from the red channel of video of a fingertip from the smartphone. This process may produce a 1D (one-dimensional) time-series signal that mimics a PPG signal. However, this signal may generally be noisier than a signal acquired from a specially designed device like a pulse oximeter and may generally have a lower sampling rate. In one implementation, a region-of-interest (e.g., the middle 200×200 pixels) may be used. In another implementation, the frames of the smartphone video may be used to create the 1D time-series PPG signal using a deep learning model. Herein, a deep learning model refers to a neural network model that utilizes one or more of the following types of layers: convolutional, residual, recurrent, attentional, and transformer. For models operating on the smartphone video frames, there may be two different types of deep learning models used: a 2D model that operates independently on each individual frame, and a 3D model that operates on two or more frames at one time. In both cases, the model produces a time-series of embeddings that may be used to produce an output, which may be another embedding, a prediction of a pulse oximeter signal, a biomarker estimation, and so on.
In one type of deep learning model, a baseline may be produced by a baseline biomarker estimation system that involves passing the averaged red channel signal or time-series of video frames through a deep learning model to predict the biomarker.
In some embodiments, a deep learning model may be trained to use smartphone data to predict a paired pulse oximeter signal. The input to the model may be the 1D time-series smartphone data, described above, from the averaging of the red channel or computed from the frames of the video. The predicted pulse oximeter signal may then be used as input to the deep learning model that will predict the biomarker.
At 604, the processor may transform the series of images into pseudo PPG signals, as described above. At 606, the processor may assess the pseudo PPG signals to determine the quality of the pseudo PPG signals. The pseudo PPG signal may have a signal characteristic associated with a quality indicator of the pseudo PPG signal. In other words, a feature of the pseudo PPG signal, such as the detailed shape of a waveform or pulse, for example, may allow for quantifying or characterizing the quality of the pseudo PPG signal. The feature may also allow for determining the circumstances (e.g., ambient conditions) that existed during acquisition of the patient physiological data. Such a feature may be compared to, or used in conjunction with, metadata gathered from measurements of light sensor 112 and accelerometer 114 during acquisition.
At 608, the processor may determine whether to provide the pseudo PPG signals to a deep learning model or to discard the pseudo PPG signals based on the determined quality. The processor may compare the signal characteristic of the pseudo PPG signal to a characteristic or parameter of a signal characteristic of a deep learning-based model PPG signal. For example, after being provided to Biomarker detection module 310, the signal characteristic of the pseudo PPG signal may be compared to a characteristic or parameter of an analogous signal characteristic of a deep learning-based model PPG signal.
At 610, the processor may measure or determine the disease state or the biomarker based on the pseudo PPG signals provided to the deep learning model, as described above.
At 708, the processor may provide to a deep learning model the pseudo PPG signals having the quality value exceeding a threshold value. In contrast, at 710, the processor may discard the pseudo PPG signals having the quality value less than the threshold value. In other words, steps 708 and 710 sort out the pseudo PPG signals according to their quality. If the quality is good, the pseudo PPG signals are used, but if their quality is bad, they are discarded. At 712, the processor may measure or determine the disease state or biomarker based on the pseudo PPG signals provided to the deep learning model.
In some implementations, in addition to measuring or determining a disease state or biomarker, an output may include displaying trends of the determined disease state or the biomarker over time. In some implementations, an application (e.g., 302) may modify a medication dosage based on the trends of the determined disease state or the biomarker. The application may then display to a user the modified medication dosage. In some cases, the application may analyze the determined disease state or the biomarker in conjunction with data based on sleep quality, stress, or anxiety levels of a patient. The display or other portion of a UI may further include an alert to a user that the determined disease state or the biomarker exceeds a preset threshold value. The display may also include a recommendation to engage in an activity or to undertake a change in behavior, and the real-time blood pressure and heart rate data.
In general, an embodiment may be implemented using a set of software instructions that are designed to be executed by a suitably programmed processing element (such as a GPU, CPU, microprocessor, processor, controller, computing device, etc.). In a complex application or system such instructions are typically arranged into “modules” with each such module typically performing a specific task, process, function, or operation. The entire set of modules may be controlled or coordinated in their operation by an operating system (OS) or other form of organizational platform.
Each application module or sub-module may correspond to a particular function, method, process, or operation that is implemented by execution of the instructions contained in the module or sub-module. Such function, method, process, or operation may include those used to implement one or more aspects, techniques, components, capabilities, steps, or stages of the described system and methods. In some embodiments, a subset of the computer-executable instructions contained in one module may be implemented by a processor in a first apparatus and a second and different subset of the instructions may be implemented by a processor in a second and different apparatus. This may happen, for example, where a process or function is implemented by steps that occur in both a client device and a remote server.
The application modules and/or sub-modules may include any suitable computer executable code or set of instructions (e.g., as would be executed by a suitably programmed processor, microprocessor, or CPU), such as computer-executable code corresponding to a programming language. For example, programming language source code may be compiled into computer-executable code. Alternatively, or in addition, the programming language may be an interpreted programming language such as a scripting language.
The modules may contain one or more sets of instructions for performing a method or function described with reference to the figures. These modules may include those illustrated but may also include a greater number or fewer number than those illustrated. As mentioned, each module may contain a set of computer-executable instructions. The set of instructions may be executed by a programmed processor contained in a server, client device, network element, system, platform, or other component.
A module may contain computer-executable instructions that are executed by a processor contained in more than one of a server, client device, network element, system, platform or other component. Thus, in some embodiments, a plurality of electronic processors, with each being part of a separate device, server, platform, or system may be responsible for executing all or a portion of the instructions contained in a specific module. Thus, although
As illustrated in
For example, Module 806 may contain computer-executable instructions which when executed by a programmed processor cause the processor or a device containing the processor to perform the functions of the client application to enable the collection and processing of the acquired data either on the device or (as an example) to navigate to a server, upload the video or other data to the server, receive the output(s) of the blood pressure model from the server, and present the output(s) to the user (in the form of a measure, graph, chart, range etc.). Module 808 may contain computer-executable instructions which when executed by a programmed processor cause the processor or a device containing the processor to generate a blood pressure detector user interface for presentation to a user. The interface may provide the user with tools and instructions for collecting the video data using their device (such as a smartphone).
The functionality for data acquisition, data processing, issue detection, and the determination of blood pressure or another health-related quantity may be provided to a user in one or more formats. These include an SDK that developers can use to perform one or more of the functions (such as data collection, initial data processing, issue detection, running the blood pressure model) in their own applications. In another embodiment, a downloaded client application is provided that is capable of performing one or more of the functions directly to an end user. In another embodiment, certain functions of the client application may be replaced by services accessible through an account on a SaaS platform.
In any of these embodiments, the trained models may reside as services on a remote server or platform so that the models can be updated and continually trained and improved as new data becomes available. This arrangement may allow anonymized data collected from multiple users to be made available to other developers and model builders for use in determining other health-related quantities and for determining correlations between medical conditions, health, and the quantities. In any of these embodiments, the interface may be provided through the web, mobile, or desktop, as long as there is an attached video camera.
Module 810 may contain computer-executable instructions which when executed by a programmed processor cause the processor or a device containing the processor to acquire the video data and subject it to processing (image and/or signal processing) to extract the red channel of the video, and to treat this signal as a pseudo-PPG signal. Module 812 may contain computer executable instructions which when executed by a programmed processor cause the processor or a device containing the processor to monitor for issues or concerns that may occur during the video/data collection and to assist in determining if the collected video/data is useable for generating the pseudo-PPG signal. Module 814 may contain computer-executable instructions which when executed by a programmed processor cause the processor or a device containing the processor to perform functions used to generate a blood pressure measure from the acquired video in cases where the Issue Detector has not prevented further processing of the video frames. As illustrated in the figure, this may include consideration of environmental conditions and signal quality prior to inputting the pseudo-PPG signal (e.g., the red channel of the video) into a trained blood pressure model. Module 815 may contain computer-executable instructions which when executed by a programmed processor cause the processor or a device containing the processor to execute additional control logic for managing the decision processes involved in obtaining device or camera parameters, setting parameters of the video/data collection process, and controlling the video/data processing flow, configuring a response to the determined blood pressure measures, etc.
In some embodiments, certain of the functionality and services provided by the system and methods described herein may be made available to multiple users by accessing an account maintained by a server or service platform. Such a server or service platform may be termed a form of Software-as-a-Service (SaaS).
It should be understood that the embodiments described above can be implemented in the form of control logic using computer software in a modular or integrated manner. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and/or methods to implement the embodiments using hardware and a combination of hardware and software.
In some embodiments, certain of the methods, models or functions described herein may be embodied in the form of a trained neural network, where the network is implemented by the execution of a set of computer-executable instructions or representation of a data structure. The instructions may be stored in (or on) a non-transitory computer-readable medium and executed by a programmed processor or processing element. The set of instructions may be conveyed to a user through a transfer of instructions or an application that executes a set of instructions (such as over a network, e.g., the Internet). The set of instructions or an application may be utilized by an end-user through access to a SaaS platform or a service provided through such a platform. A trained neural network, trained machine learning model, or other form of decision or classification process may be used to implement one or more of the methods, functions, processes, or operations described herein. Note that a neural network or deep learning model may be characterized in the form of a data structure in which are stored data representing a set of layers containing nodes, and connections between nodes in different layers are created (or formed) that operate on an input to provide a decision or value as an output.
In general terms, a neural network may be viewed as a system of interconnected artificial “neurons” that exchange messages between each other. The connections have numeric weights that are “tuned” during a training process, so that a properly trained network will respond correctly when presented with an image or pattern to recognize (for example). In this characterization, the network consists of multiple layers of feature-detecting “neurons”; each layer has neurons that respond to different combinations of inputs from the previous layers. Training of a network is performed using a “labeled” dataset of inputs in a wide assortment of representative input patterns that are associated with their intended output response. Training uses general-purpose methods to iteratively determine the weights for intermediate and final feature neurons. In terms of a computational model, each neuron calculates the dot product of inputs and weights, adds the bias, and applies a non-linear trigger or activation function (for example, using a sigmoid response function).
Machine learning (ML) is being used more and more to enable the analysis of data and assist in making decisions in multiple industries. In order to benefit from using machine learning, a machine learning algorithm is applied to a set of training data and labels to generate a “model” which represents what the application of the algorithm has “learned” from the training data. Each element (or example, in the form of one or more parameters, variables, characteristics or “features”) of the set of training data is associated with a label or annotation that defines how the element should be classified by the trained model. A machine learning model is a set of layers of connected neurons that operate to make a decision (such as a classification) regarding a sample of input data. When trained (i.e., the weights connecting neurons have converged and become stable or within an acceptable amount of variation), the model will operate on a new element of input data to generate the correct label or classification as an output.
An embodiment, which is provided as a particular example and is not intended to exclude other suitable methods, includes a methodology for training a biomarker estimation deep learning model, which may be trained on a large bank of biosignals (e.g., collected from public and private datasets). This may include PPG, ECG, seismocardiogram (SCG), ballistocardiogram (BCG), Electroencephalogram (EEG), and more. The output of the model may be a series of embeddings that are utilized to predict some downstream task.
The training may utilize both unsupervised and semi-supervised learning. For unsupervised, portions of the signal may be removed from the input sequence, and the model may be instructed to predict the missing portion of embeddings. In semi-supervised, certain characteristics of the signal may be calculated (such as heart rate from the PPG signal), for which the model will attempt to predict these portions. In another iteration of semi-supervised, certain characteristics about samples that are similar will be calculated (e.g., from the same person, have the same heart rate, etc.) and the model will be tasked with creating an embedding that is similar for these samples with the same characteristics.
The embeddings created by the model may then be used for biomarker estimation. A deep learning model may operate on the embeddings created by the large pre-trained model to predict the biomarker, which may be systolic BP and diastolic BP, for example.
For the large pretrained model, one may quantize the different values of the input signal. For example, for a signal that can take values of 0 to 1, one may create 1 million equally spaced states, where each value of the signal is rounded to the nearest state.
As discussed above, there may be multiple sources of time-synchronized sensor data collected at one time. For example, sources may include the pulse oximeter, IMU, ECG, or the smartphone optical and motion signals. These signals may be combined as separate channels of the input signal (e.g., channel 0=PPG, channel 1=accelerometer X, channel 2=accelerometer Y). The deep learning model would then operate on this multi-channel signal.
The deep learning model may be used to predict not only the biomarker of interest, but also additional features about the signal. For example, heart rate may be calculated from the PPG signal. The deep learning model would be instructed to predict both heart rate and the biomarker of interest. The prediction of these additional features can improve the performance of the biomarker prediction.
One such example is the prediction of pulse transit time (PTT). Using an IMU signal (motion) and optical signal, the time interval from when the aortic opening is sensed by the IMU and a landmark point in the optical signal (e.g., peak, valley, beginning of rise to systolic peak) may be measured. This time interval is PTT, which may be predicted by a model that contains both IMU and optical data. One may train different machine learning models for different demographics. For example, one may train a model for individuals of ages 0-40, 40-60, and 60+. During inference, one would use the model that matches the individual's demographics.
The data collected from the ear and toe pulse oximeters may be used to calculate the PTT. By using a landmark point in both signals (e.g., peak, valley, location of rise to systolic peak, etc.), the time interval between the landmark point in the ear and toe signals is calculated. Next, a deep learning model running on either just the optical fingertip signal or optical fingertip and IMU data may be used to predict that PTT value.
An embodiment, which is provided as a particular example and is not intended to exclude other suitable methods, includes a methodology for collecting training data for a model or models. A data collection process may include, for example, a data collection person or persons. They first guide a subject through a relaxation period, such as about five minutes, which may allow for stabilizing the subject's blood pressure. Then they may take blood pressure readings (discarding the first) and alternative blood pressure readings and fingertip videos on a variety of smartphones. Throughout this process, the subject may be connected to a pulse oximeter device to capture ground truth PPG. They also may take demographic information that may affect the reading, including age, height, weight, biological sex, hand size, and skin tone. To ensure accuracy of the BP reading, BP may be measured via auscultation and a digital stethoscope may be used along with a webcam. The webcam video may be spliced with stethoscope audio. A data collection application may record fingertip videos. This application may record fingertip video with frame-by-frame metadata that includes the current camera settings for each frame. To align the PPG reading and a smartphone fingertip reading, a breadboard that has a button hooked up to an LED (e.g., a red LED) and to the pulse oximeter may be used. The button activates the LED and sends a signal to the PPG device to start recording. The smartphone is held up to the LED and the smartphone and PPG signals may be aligned based on that light. The data collection process may be mobile to enable capture of training data from a greater diversity of subjects by going to them on-site rather than them having to come to a specific location. Once data is collected, it may be uploaded into cloud storage. Processing the data may begin once it's uploaded, automatically syncing the PPG and smartphone fingertip reading (by looking for a spike in the red), clipping the smartphone and PPG signals to just the relevant portions with the fingertip on the camera (by looking for ratio of red to blue and green), fine-tuning the alignment of the PPG and smartphone signal, scoring the smartphone signals, and producing plots that can be analyzed. This data can be used to train the model(s). In addition to preparing data and producing plots, a database may also be produced of individual pulses that include signal quality score info and demographic data that we can use to debug a model quickly. The system may potentially use the demographic data as input into the BP model for greater accuracy as well, and for segmenting users to provide more accurate BP values.
In some embodiments, the system or services described herein may be implemented as microservices, processes, workflows or functions performed in response to the submission of a set of input data. The microservices, processes, workflows or functions may be performed by a server, data processing element, platform, or system. In some embodiments, the data analysis and other services may be provided by a service platform located “in the cloud”. In such embodiments, the platform may be accessible through APIs and SDKs. The functions, processes and capabilities described herein and with reference to one or more of the figures may be provided as microservices within the platform. The interfaces to the microservices may be defined by REST and GraphQL endpoints. An administrative console may allow users or an administrator to securely access the underlying request and response data, manage accounts and access, and in some cases, modify the processing workflow or configuration.
Note that although some embodiments described herein are directed to a multi-tenant or SaaS architecture that may be used for the delivery of business-related or other applications and services to multiple accounts/users, such an architecture may also be used to deliver other types of data processing services and provide access to other applications. For example, such an architecture may be used to provide one or more of the processes, functions, and operations described herein. Although in some embodiments, a platform or system of the type illustrated in the figures may be operated by a 3rd party provider to provide a specific set of services or applications, in other embodiments, the platform may be operated by a provider and a different entity may provide the applications or services for users through the platform.
System 900, which may be hosted by a third party, may include a set of data analysis and other services to assist in acquiring and processing video data, and generating a measure or measures of one or more health-related quantities, and a web interface server 914. The services may include one or more functions or operations for the processing of acquired video or other data to enable the generation of a measure of blood pressure or other health-related quantity. As examples, in some embodiments, the set of functions, operations or services made available through the platform or system 900 may include Account Management services 916, which may implement a process or service to authenticate a user wishing to submit an example of video data and from that determine a measure of a health-related quantity, or which may implement a process or service to generate a container or instantiation of the data processing and analysis services for that user. System 900 may also include Initial Data Acquisition and Processing services 917, which may implement a process or service to provide a client-side application to a user for installation on their device, may implement a process or service to receive a set of frames of video data from the user device; or may implement a process or service to process each frame to extract a red or pseudo-PPG channel (the PPG Signal Generator)
System 900 may also include Identify/Detect Video Collection Issues of Possible Concern services 918, which may implement a process or service to monitor and detect one or more situations or events that might indicate the video data acquired during that situation or event is not reliable or accurate, or may implement a process or service to execute logic to control the collection or processing of the video frames based on the presence or absence of such a situation or event. System 900 may also include Determine Blood Pressure Measure(s) and/or Other Health-Related Quantities services 919, which may implement a process or service to assess environmental factors and signal quality factors (in one embodiment using a trained signal quality assessment model), and if those are acceptable, to provide the generated (extracted) PPG signal to a trained model. Determine Blood Pressure Measure(s) and/or Other Health-Related Quantities services 919 may also implement a trained model that operates to receive the generated PPG signal extracted from the video frames and in response to generate a measure or measures of one or more health related quantities, such as blood pressure.
System 900 may also include Control Logic services 920, which may implement a process or service to implement other needed control logic to control the acquisition, processing, and use of the trained model(s), may implement a process or service to implement other logic for delivering and presenting the output of the trained blood pressure model to a user, including trend data and/or other forms of analysis of blood pressure measures, or may implement a process or service to implement logic to process data from other sensors and/or to process such data to generate other forms of signals for input to a trained model. System 900 may also include administrative services 922, which may implement a process or services to provide platform and services administration, such as, for example, enabling the provider of the services and/or the platform to administer and configure the processes and services provided to users.
The platform or system illustrated in
The distributed computing service/platform (which may also be referred to as a multi-tenant data processing platform) 1008 may include multiple processing tiers, including a user interface tier 1016, an application server tier 1020, and a data storage tier 1024. The user interface tier 1016 may maintain multiple user interfaces 1017, including graphical user interfaces and/or web-based interfaces. The user interfaces may include a default user interface for the service to provide access to applications and data for a user or “tenant” of the service (depicted as “Service UI” in the figure), as well as one or more user interfaces that have been specialized/customized in accordance with user specific requirements (e.g., represented by “Tenant A UI”, . . . , “Tenant Z UI” in the figure, and which may be accessed via one or more APIs).
The default user interface may include user interface components enabling a tenant to administer the tenant's access to and use of the functions and capabilities provided by the service platform. This may include accessing tenant data, launching an instantiation of a specific application, causing the execution of specific data processing operations, etc. Each application server or processing tier 1022 illustrated in the figure may be implemented with a set of computers and/or components including computer servers and processors, and may perform various functions, methods, processes, or operations as determined by the execution of a software application or set of instructions. The data storage tier 1024 may include one or more data stores, which may include a Service Data store 1025 and one or more Tenant Data stores 1026. Data stores may be implemented with any suitable data storage technology, including structured query language (SQL) based relational database management systems (RDBMS).
Service Platform 1008 may be multi-tenant and may be operated by an entity to provide multiple tenants with a set of business-related or other data processing applications, data storage, and functionality. For example, the applications and functionality may include providing web-based access to the functionality used by a business to provide services to end-users, thereby allowing a user with a browser and an Internet or intranet connection to view, enter, process, or modify certain types of information. Such functions or applications are typically implemented by one or more modules of software code/instructions that are maintained on and executed by one or more servers 1022 that are part of the platform's Application Server Tier 1020. As noted with regards to
As mentioned, rather than build and maintain such a platform or system themselves, a business may utilize systems provided by a third party. A third party may implement a business system/platform as described above in the context of a multi-tenant platform, where individual instantiations of a business' data processing workflow (such as the data analysis and evaluation services and processing described herein) are provided to users, with each business representing a tenant of the platform. One advantage to such multi-tenant platforms is the ability for each tenant to customize their instantiation of the data processing workflow to that tenant's specific business needs or operational methods. Each tenant may be a business or entity that uses the multi-tenant platform to provide business services and functionality to multiple users.
As noted,
The application layer 1110 may include one or more application modules 1111, each having one or more sub-modules 1112. Each application module 1111 or sub-module 1112 may correspond to a function, method, process, or operation that is implemented by the module or sub-module (e.g., a function or process related to providing business related data processing and services to a user of the platform). Such function, method, process, or operation may include those used to implement one or more aspects of the inventive system and methods, such as for one or more of the processes or functions described with reference to the figures.
The application modules and/or sub-modules may include any suitable computer-executable code or set of instructions (e.g., as would be executed by a suitably programmed processor, microprocessor, or CPU), such as computer-executable code corresponding to a programming language. For example, programming language source code may be compiled into computer-executable code. Alternatively, or in addition, the programming language may be an interpreted programming language such as a scripting language. Each application server (e.g., as represented by element 1022 of
The data storage layer 1120 may include one or more data objects 1122 each having one or more data object components 1121, such as attributes and/or behaviors. For example, the data objects may correspond to tables of a relational database, and the data object components may correspond to columns or fields of such tables. Alternatively, or in addition, the data objects may correspond to data records having fields and associated services. Alternatively, or in addition, the data objects may correspond to persistent instances of programmatic data objects, such as structures and classes. Each data store in the data storage layer may include each data object. Alternatively, different data stores may include different sets of data objects. Such sets may be disjoint or overlapping.
Note that the example computing environments depicted in
Any of the software components, processes or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as Python, Java, JavaScript, C++ or Perl using conventional or object oriented techniques. The software code may be stored as a series of instructions, or commands in (or on) a non-transitory computer-readable medium, such as a random-access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM. In this context, a non-transitory computer-readable medium is almost any medium suitable for the storage of data or an instruction set aside from a transitory waveform. Any such computer readable medium may reside on or within a single computational apparatus and may be present on or within different computational apparatuses within a system or network.
According to one example implementation, the term processing element or processor, as used herein, may be a central processing unit (CPU), or conceptualized as a CPU (such as a virtual machine). In this example implementation, the CPU or a device in which the CPU is incorporated may be coupled, connected, and/or in communication with one or more peripheral devices, such as display. In another example implementation, the processing element or processor may be incorporated into a mobile computing device, such as a smartphone or tablet computer.
The non-transitory computer-readable storage medium referred to herein may include a number of physical drive units, such as a redundant array of independent disks (RAID), a floppy disk drive, a flash memory, a USB flash drive, an external hard disk drive, thumb drive, pen drive, key drive, a High-Density Digital Versatile Disc (HD-DVD) optical disc drive, an internal hard disk drive, a Blu-Ray optical disc drive, or a Holographic Digital Data Storage (HDDS) optical disc drive, synchronous dynamic random access memory (SDRAM), or similar devices or other forms of memories based on similar technologies. Such computer-readable storage media allow the processing element or processor to access computer-executable process steps, application programs and the like, stored on removable and non-removable memory media, to off-load data from a device or to upload data to a device. As mentioned, with regards to the embodiments described herein, a non-transitory computer-readable medium may include almost any structure, technology, or method apart from a transitory waveform or similar medium.
Certain implementations of the disclosed technology are described herein with reference to block diagrams of systems, and/or to flowcharts or flow diagrams of functions, operations, processes, or methods. It will be understood that one or more blocks of the block diagrams, or one or more stages or steps of the flowcharts or flow diagrams, and combinations of blocks in the block diagrams and stages or steps of the flowcharts or flow diagrams, respectively, may be implemented by computer-executable program instructions. Note that in some embodiments, one or more of the blocks, or stages or steps may not necessarily need to be performed in the order presented or may not necessarily need to be performed at all.
These computer-executable program instructions may be loaded onto a general purpose computer, a special purpose computer, a processor, or other programmable data processing apparatus to produce a specific example of a machine, such that the instructions that are executed by the computer, processor, or other programmable data processing apparatus create means for implementing one or more of the functions, operations, processes, or methods described herein. These computer program instructions may also be stored in a computer readable memory that may direct a computer or other programmable data processing apparatus to function in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement one or more of the functions, operations, processes, or methods described herein.
Other objects and advantages of the systems and methods described will be apparent to one of ordinary skill in the art upon review of the detailed description and the included figures. Throughout the drawings, identical reference characters and descriptions indicate similar, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been illustrated by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.
The use of the terms “a” and “an” and “the” and similar referents in the specification and in the following claims are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “having,” “including,” “containing” and similar referents in the specification and in the following claims are to be construed as open-ended terms (e.g., meaning “including, but not limited to,”) unless otherwise noted. Recitation of ranges of values herein are merely indented to serve as a shorthand method of referring individually to each separate value inclusively falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein may be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate embodiments and does not pose a limitation to the scope of the embodiments unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to each embodiment.
As used herein (e.g., the claims, figures, and specification), the term “or” is used inclusively to refer items in the alternative and in combination.
The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the disclosure. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the systems and methods described herein. The foregoing descriptions of specific embodiments or examples are presented by way of examples for purposes of illustration and description. They are not intended to be exhaustive of or to limit this disclosure to the precise forms described. Many modifications and variations are possible in view of the above teachings. The embodiments or examples are illustrated and described in order to best explain the principles of this disclosure and practical applications, to thereby enable others skilled in the art to best utilize this disclosure and various embodiments or examples with various modifications as are suited to the particular use contemplated. It is intended that the scope of this disclosure be defined by the following claims and their equivalents.
Claims
1. A method of measuring or determining a disease state or a biomarker, the method comprising:
- acquiring a series of images of a tissue using a camera of a mobile device;
- transforming the series of images into pseudo photoplethysmography (PPG) signals;
- assessing the pseudo PPG signals to determine the quality of the pseudo PPG signals;
- determining whether to provide the pseudo PPG signals to a deep learning model or to discard the pseudo PPG signals based on the determined quality; and
- measuring or determining the disease state or the biomarker based on the pseudo PPG signals provided to the deep learning model.
2. The method of claim 1, wherein the series of images comprises a video signal.
3. The method of claim 1, wherein acquiring the series of images includes controlling a flash and exposure settings of the camera.
4. The method of claim 1, wherein discarding the pseudo PPG signals comprises:
- not providing the pseudo PPG signals to the deep learning model;
- adjusting the flash or the exposure settings of the camera; and
- initiating additional acquiring of images of the tissue using the adjusted flash or exposure settings.
5. The method of claim 1, further comprising:
- measuring ambient light reaching the tissue using a photosensor onboard the mobile device;
- measuring relative motion between the tissue and the camera using an accelerometer onboard the mobile device; and
- grouping the measured ambient light and the measured relative motion as metadata.
6. The method of claim 5, wherein the quality of the pseudo PPG signals is based, at least in part, on the metadata.
7. The method of claim 1, wherein assessing the pseudo PPG signals to determine the quality of the pseudo PPG signals further comprises analyzing shapes and features of waveforms of the pseudo PPG signals, wherein the quality of the pseudo PPG signals is based, at least in part, on the shapes and features of the waveforms.
8. The method of claim 1, wherein the biomarker comprises measures of blood pressure, heart rate, SpO2, or other health-related quantity.
9. The method of claim 1, wherein the mobile device is a smartphone, wrist-worn wearable such as smartwatch or smart wristband, earbud, or tablet.
10. A method of measuring or determining a disease state or biomarker, the method comprising:
- acquiring a series of images of a tissue using a camera of a mobile device;
- transforming the series of images into pseudo photoplethysmography (PPG) signals;
- assigning a quality value to the pseudo PPG signals based on the quality of the pseudo PPG signals;
- providing to a deep learning model the pseudo PPG signals having the quality value exceeding a threshold value;
- discarding the pseudo PPG signals having the quality value less than the threshold value; and
- measuring or determining the disease state or biomarker based on the pseudo PPG signals provided to the deep learning model.
11. The method of claim 10, wherein transforming the series of images into the PPG signals comprises applying a second deep learning model to the series of images, wherein an output of the second deep learning model is the PPG signals.
12. The method of claim 10, wherein the series of images comprises a video signal.
13. The method of claim 10, wherein acquiring the series of images includes controlling a flash and exposure settings of the camera.
14. The method of claim 13, wherein discarding the pseudo PPG signals comprises:
- not providing the pseudo PPG signals to the deep learning model;
- adjusting the flash or the exposure settings of the camera; and
- initiating additional acquiring of images of the tissue using the adjusted flash or exposure settings.
15. The method of claim 10, further comprising:
- measuring ambient light reaching the tissue using a photosensor onboard the mobile device;
- measuring relative motion between the tissue and the camera using an accelerometer onboard the mobile device; and
- grouping the measured ambient light and the measured relative motion as metadata.
16. The method of claim 15, wherein the quality of the pseudo PPG signals is based, at least in part, on the metadata.
17. The method of claim 10, wherein assessing the pseudo PPG signals to determine the quality of the pseudo PPG signals further comprises analyzing shapes and features of waveforms of the pseudo PPG signals, wherein the quality of the pseudo PPG signals is based, at least in part, on the shapes and features of the waveforms.
18. The method of claim 10, wherein the biomarker comprises measures of blood pressure and heart rate.
19. The method of claim 10, wherein the biomarker comprises measures SpO2 or other health-related quantity.
20. The method of claim 10, wherein the mobile device is a smartphone, wrist-worn wearable such as smartwatch or smart wristband, earbud, or tablet.
Type: Application
Filed: Jul 28, 2022
Publication Date: Feb 2, 2023
Inventors: Matthew Steven Whitehill (Seattle, WA), Eric Chen (Seattle, WA), Jamien McCullum (Seattle, WA), Jessie Young (Seattle, WA)
Application Number: 17/815,901
