MACHINE LEARNING-BASED PREDICTION OF PHYSIOLOGICAL PARAMETERS IN REMOTE MEDICAL INFORMATION EXCHANGE

Abstract

System and method for remote medical information exchange are disclosed. The system comprises a patient-end digital device configured to capture video stream of face of the patient. The system comprises a doctor-end digital device configured to show the video stream of the face of the patient and one or more physiological inference in accordance with the video stream. The system comprises a processor configured to receive a facial image from the captured video stream. The processor can provide the facial image to a trained neural network configured to predict a sentiment of the patient based on features in the facial image. The processor is configured to receive the sentiment of the patient from the trained neural network indicating how the patient is feeling. The processor is further configured to provide the predicted sentiment to the doctor-end digital device to aid in diagnosis of the patient.

Description

PRIORITY APPLICATIONS

This application is a continuation of U.S. Non-Provisional patent application Ser. No. 16/802,485, titled, “SYSTEM AND METHOD FOR REMOTE MEDICAL INFORMATION EXCHANGE,” filed Feb. 26, 2020 (Attorney Docket No. DCAI 1007-2), which claims priority to or the benefit of U.S. Provisional Patent Application Nos. 62/883,070 titled, “ACCELERATED PROCESSING OF GENOMIC DATA AND STREAMLINED VISUALIZATION OF GENOMIC INSIGHTS,” filed Aug. 5, 2019, (Attorney Docket No. DCAI 1000-1); U.S. Provisional Patent Application No. 62/942,644, titled, “SYSTEMS AND METHODS OF TRAINING PROCESSING ENGINES,” filed Dec. 2, 2019, (Attorney Docket No. DCAI 1002-1); U.S. Provisional Patent Application No. 62/975,177, filed Feb. 11, 2020, titled, “ARTIFICIAL INTELLIGENCE-BASED DRUG ADHERENCE MANAGEMENT AND PHARMACOVIGILANCE,” (Attorney Docket No. DCAI 1005-1); U.S. Provisional Patent Application No. 62/810,549, titled, “SYSTEM AND METHOD FOR REMOTE MEDICAL INFORMATION EXCHANGE,” filed Feb. 26, 2019 (Attorney Docket No. DCAI 1007-1);

This application is a continuation of U.S. patent application Ser. No. 16/802,485, which is a continuation-in-part of U.S. patent application Ser. No. 15/946,629, entitled “IMAGE-BASED SYSTEM AND METHOD FOR PREDICTING PHYSIOLOGICAL PARAMETERS,” filed on Apr. 5, 2018 (Atty. Docket No. DCAI 1006-2), which claims the benefit of U.S. Provisional Application No. 62/481,691, entitled “METHOD OF BODY MASS INDEX PREDICTION BASED ON SELFIE IMAGES,” filed on Apr. 5, 2017 (Atty. Docket No. DCAI 1006-1). The provisional and non-provisional applications are hereby incorporated by reference for all purposes.

INCORPORATIONS

The following materials are incorporated by reference as if fully set forth herein:

U.S. Provisional Patent Application No. 62/883,639, titled “FEDERATED CLOUD LEARNING SYSTEM AND METHOD,” filed on Aug. 6, 2019 (Atty. Docket No. 396892-991101);

U.S. Provisional Patent Application No. 62/816,880, titled “SYSTEM AND METHOD WITH FEDERATED LEARNING MODEL FOR MEDICAL RESEARCH APPLICATIONS,” filed on Mar. 11, 2019;

U.S. Provisional Patent Application No. 62/671,823, titled “SYSTEM AND METHOD FOR MEDICAL INFORMATION EXCHANGE ENABLED BY CRYPTO ASSET,” filed on May 15, 2018; and

U.S. Nonprovisional patent application Ser. No. 16/167,338, titled “SYSTEM AND METHOD FOR DISTRIBUTED RETRIEVAL OF PROFILE DATA AND RULE-BASED DISTRIBUTION ON A NETWORK TO MODELING NODES,” filed on Oct. 22, 2018.

TECHNICAL FIELD

The disclosed system and method are in the field of information technologies to exchange medical information remotely.

BACKGROUND

Recent years, patients and plain consumers are getting more and more educated about personal health care and medical information. Generations with smartphones, which is almost all population in developed countries and growing population in developing countries, are enabled to check, collect, gather medical and health care information and data like never before. On the other hand, health and medical companies are having issues scaling their services, there are not enough physicians and family doctors, there are not enough nurses, and there are not enough genetic counselors. Existing technologies are ready for us to help health and medical services suppliers to scale customer supports. The strong development in artificial intelligence in natural language understanding and processing makes it possible for us to teach our machines augment medical and health care services in a dramatic way.

Telehealth, i.e., health care services via telecommunications or similar, is becoming a popular alternative to the make up for the lack of general practitioners, genetic counsellors, and in general to scale care.

Most of the Telehealth solutions today require doctors to make a vision inspection patient over a video feed and infer all the medical information via a dialog with the patient remotely. There are a few setbacks that make Telehealth not the most efficient form of care.

One of the setbacks is that doctors do not have the same level of visibility to a patient's facial expression, feeling, or sentiments as compared to in person. A lot of diagnosis depends on how the patient looks, feels, and behaves that physicians are trained to read as cues for diagnostics. Secondly, doctors can miss asking relevant question as the necessary context can be missing in a remote set-up. Thirdly, it is also harder for patients to provide all the requisite medical history including modern type of data like genetic data, microbiome data, or lab results to the doctor effectively over a video feed. Therefore, in the end, the diagnostics provided over Telehealth is not as effective as in-person consultation.

In the present disclosure, system and method for remote medical information exchange are to be discussed.

SUMMARY

Generally provided are a system and method for remote medical information exchanging by a telehealth augmentation platform. The system for medical information exchanging can be used directly via phenomenal face mobile application program as well as embedded into other compatible platforms that support telehealth. With Phenomenal Telehealth Augmentation doctors are provided auto inferred medical information about the patient while having the video conversation. This enables the doctors and physicians to make better decision and converge to the right conversations quickly in order to provide better diagnostics. Some examples of the type of inference that is provided during the video feed can include auto inference of age, sex, height, weight, auto inference of mood and sentiment to reflect how the patient feels, auto inference of recommended screenings and diagnostics, auto inference of exposome data like air quality, pollen exposure, socio-economic risks, etc. based on the location from where the user is connecting, auto inference of risks for genetic diseases if the patient has uploaded genetic data, auto inference of past medical records and past prescriptions using camera to capture this information.

This summary is provided to efficiently present the general concept of the invention and should not be interpreted as limiting the scope of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For purpose of facilitating understanding of the embodiments, the accompanying drawings and description illustrate embodiments thereof, its various structures, construction, method of operation, and many advantages that may be understood and appreciated. According to common practice, the various features of the drawings are not drawn to scale. To the contrary, the dimensions of the various features are expanded or reduced for the purpose of explanation and clarity.

FIG. 1 is a diagram illustrating an exemplary information exchange scenario for the applications of healthcare services and clinical and/or data trials, consistent with embodiments of the present disclosure.

FIG. 2 is a diagram illustrating an exemplary data storage center with cloud services and its machine learning modules, as well as the interaction between live video stream with these machine learning modules, consistent with embodiments of the present disclosure.

FIG. 3 is a diagram illustrating an exemplary system architecture of data storage center with cloud services, consistent with embodiments of the present disclosure.

DETAILED DESCRIPTION

Many alternative embodiments of the present aspects may be appropriate and are contemplated, including as described in these detailed embodiments, though also including alternatives that may not be expressly shown or described herein but as obvious variants or obviously contemplated according to one of ordinary skill based on reviewing the totality of this disclosure in combination with other available information. For example, it is contemplated that features shown and described with respect to one or more embodiments may also be included in combination with another embodiment even though not expressly shown and described in that specific combination.

For purpose of efficiency, reference numbers may be repeated between figures where they are intended to represent similar features between otherwise varied embodiments, though those features may also incorporate certain differences between embodiments if and to the extent specified as such or otherwise apparent to one of ordinary skill, such as differences clearly shown between them in the respective figures.

Reference is now made to FIG. 1, which is a diagram illustrating an exemplary information exchange scenario for the applications of healthcare services and clinical and/or data trials, consistent with embodiments of the present disclosure.

Patient or end user uses a phenomenal telehealth application program to connect to a virtual telehealth consultation session. The phenomenal telehealth application can be installed at an electronic device 110 at the patient or end user's end, a mobile device, e.g. a mobile phone, a tablet, etc., and can also be installed at a computer connected to internet or a secured network. A doctor or physician uses the application program to connect to the virtual telehealth consultation session, from the doctor or physician's end via an electronic device 120, e.g., a mobile device or ideally a computer connected to internet or a secured network. In some embodiments, the phenomenal telehealth application can be integrated and embedded into other third party telehealth application or platform using a telehealth license module. The phenomenal telehealth application does not necessarily run on its own or stand alone.

In some embodiments, medical or health related data and information of the patient or end user can be received by the exchange platform and stored in a remote and secured data storage center with cloud services 130, e.g., Google Cloud, Amazon AWS, Microsoft Azure Cloud, or other cloud services. There are one or more auto inference machine learning modules preinstalled in data storage center with cloud services 130.

At the patient or end user's end, patient or end user's face is to be captured by camera of electronic device 110 in a live video form. Via data storage center with cloud services 130 and communication network, the live video stream is to be sent to the doctor or physician's end and shown on screen of electronic device 120. At the same time, on the screen of electronic device 120, the inference results derived by auto inference machine learning modules are shown at another portion of the screen.

In some embodiments, the inference results can include various physiological parameters of the patient or end user undergoing the virtual telehealth consultation session. These physiological parameters include but not limited to age, height, weight, sex, screening, mood, risks, etc. All the physiological parameters are real-time derived and reflect the individual information of the patient or end user. Such physiological parameters are augment to the facial live video stream. The inference information significantly reduces the time required for the doctor to understand the patient's basic essentials as well as other information that might be missed due to the lack of in person context. This will also drive the conversation to more of an individualized feeling level and focus on the patient's problem instead of wasting time on less essential information.

Reference is now made to FIG. 2, which is a diagram illustrating an exemplary data storage center with cloud services and its machine learning modules, as well as the interaction between live video stream with these machine learning modules, consistent with embodiments of the present disclosure.

When the live video stream is streamed up to data storage center with cloud services 130, machine learning modules pre-installed in the cloud start analyzing the live video stream. In some embodiments, machine learning module can include but not limited to phenome artificial intelligence module 201, exposome artificial intelligence module 202, real-time Global Position System (GPS) or navigation module 203, genetics and bioinformatics module 204, reverse synthetic Pharmacy Benefit Manager (PBM) module 205, MedVision module 206, Manna food module 207, predictions machine module 208, etc. The pre-installed machine learning modules can be one or more of any combination of the aforementioned modules. Each machine learning module functions as its name indicates. A combined and cross-referenced inference physiological information is to be streamed to electronic device 120 at the doctor or physician's end and shown on the screen.

Reference is now made to FIG. 3, which is a diagram illustrating an exemplary system architecture of data storage center with cloud services, consistent with embodiments of the present disclosure.

The system is largely a composable set of backend artificial intelligence-based modules that uses data from software development kit and/or from telehealth application program to make a combined prediction for virtual consultation session. In some embodiments, the system services comprise scalable video server 301, phenome artificial intelligence inference module 201, real-time GPS module 203, exposome artificial intelligence inference module 202, genetics and bioinformatics module 204, reverse synthetic PBM module, MedVision module, Manna food module, poly-omics uni-matrix pipeline 302, and combined inference module 303.

Scalable video server 301 ingests the live video stream of the patient or end user during the virtual consultation session for interpretations by doctor or physician.

Scalable video server 301 is configured to ensure the user's privacy is preserved and it is via an opt-in with consent from the patient or end user. It is also configured to high security to facilitate one on one video conversation. Phenome artificial intelligence inference module 201 is configured to extract facial data and infers physiological parameters including age, sex, height, weight, BMI, mood, life-expectancy, or other statistical measures. Real-time GPS module 203 is configured to constantly synchronize to the patient or end user's physical location and past locations in order to understand the exposimics of the user. Exposome artificial intelligence inference module 202 is configured to compute the risks and exposures to air quality, pollen, walkability (how active one is allowed in the condition), socio-economic risks, etc. Inference data from this module augments the risk for certain medical conditions, e.g., Asthma, allergies, etc., that a doctor or physician should be aware of. Further description of these modules is found in ARTIFICIAL INTELLIGENCE-BASED DRUG ADHERENCE MANAGEMENT AND PHARMACOVIGILANCE, incorporated above by reference.

Genetics and bioinformatics module 204 is configured to allow a patient or end user upload his/her genetic profile data, from consumer faced genetic services such as 23andMe, Ancestry, full exome sequence, whole genome sequence, etc., as part of the application program onboarding procedure. Further description of handling and analysis of genetic information if found in ACCELERATED PROCESSING OF GENOMIC DATA AND STREAMLINED VISUALIZATION OF GENOMIC INSIGHTS and ARTIFICIAL INTELLIGENCE-BASED DRUG ADHERENCE MANAGEMENT AND PHARMACOVIGILANCE, incorporated above by reference. The information collected by genetic and bioinformatics module can be used along with the combination of other phenotypic data from phenome artificial intelligence inference module 201. Reverse synthetic pharmacy benefit manager (PBM) module is configured to allow patient or end user to optionally add medications and prescription information using the MedVision module. Further description of PBM and MedVision modules is found in ARTIFICIAL INTELLIGENCE-BASED DRUG ADHERENCE MANAGEMENT AND PHARMACOVIGILANCE, incorporated above by reference. The reverse synthetic PBM module curates the drug information for patient or end user. Such drug information can also be cross-referenced with genetics data to red flag risks of certain drugs to patient or end user.

MedVision module is configured to allow patient or end user to import his/her medical records and medication information by using the camera of electronic device 110. Patient and end user can use the cameral of electronic device 110 to take a picture over medications he/she takes. MedVision module 206 is configured to process the image to capture the medication information. Manna food module 207 is configured to estimate the probability that food in a picture is scientifically considered to be healthy or not. Polyomics uni-matrix pipeline 302 is configured to be fed by data from some or all aforementioned artificial intelligence modules. It joins multiple omics collected to build predictive values on the patient or end user in a real-time manner. Further description of polyomics is found in ARTIFICIAL INTELLIGENCE-BASED DRUG ADHERENCE MANAGEMENT AND PHARMACOVIGILANCE, incorporated above by reference. Combined inference module 303 is configured to combine all the predictions and inferences from some or all modules. Display feed data formed by combined inference module 303 is to be passed back to doctor's end of phenomenal telehealth application.

Although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications can be made in the details within the scope of equivalents of the claims by anyone skill in the art without departing from the invention.

The ability for a doctor or physician to get auto inferred information while keeping the patient or end user highly engaged on a video stream is a groundbreaking change for Telehealth. It allows doctors and physicians to focus on having scientifically and medically more meaningful conversation with patients or end users on medical problems instead of frivolous data collection.

Most importantly, as patient or end user adds additional data into the phenomenal telehealth application, the information is correlated and used for building better predictions. The confluence of all data from end user and context means that doctor or physician doesn't have to do administrative work of data collection during a patient's consultation, thus saving a lot of time and being more effective providing diagnostics.

In the following claim, the modules can be used by themselves, in pairs, in triplets and in other combinations. Not all of the modules need to be used to compose a functional and beneficial system. This applies as well to methods and CRMs that practice the technology disclosed.

The technology disclosed also can be practiced as a method or non-transitory computer readable medium. As a method, the technology disclosed exercises the functions indicated by the various modules, as described herein and in the materials incorporated by reference. As a CRM, the technology disclosed includes computer program instructions that, when combined with computer hardware, result in a system as described with various modules, as described herein and in the materials incorporated by reference. Alternatively, as a CRM, the technology disclosed includes computer program instructions that, when executed on one or more processors coupled to memory, displays, a network interface, etc., exercises the functions indicated by the various modules, as described herein and in the materials incorporated by reference.

Claims

1. A system for remote medical information exchange, comprising:

a patient-end digital device, configured to capture video stream of face of a patient;

a doctor-end digital device, configured to show the video stream of the face of the patient, and one or more physiological inference in accordance with the video stream;

a processor, electrically coupled with a communication network, configured to: receive from the patient-end digital device a facial image from the captured video stream including facial and upper body features of the patient; process the facial image to generate a frontal facial image; provide the frontal facial image, comprising facial and upper body features, to a first trained neural network model configured to predict a sentiment of the patient based on the facial and upper body features; receive the sentiment of the patient from the first trained neural network model indicating how the patient is feeling; wherein the first trained neural network model is a regression deep learning convolutional neural network model; and provide the predicted sentiment of the patient as at least one of the one or more physiological inference to the doctor-end digital device.

2. The system of claim 1, wherein the first trained neural network model has a plurality of input parameters, the input parameters including three color channels corresponding to one or more images.

3. The system of claim 1, wherein the regression deep learning convolutional neural network model is a Network-in-Network neural network model.

4. The system of claim 1, wherein the frontal facial image is provided via three color channels.

5. The system of claim 1, further comprising a server, electrically coupled with the communication network, and wherein the first trained neural network model is stored on the server.

6. The system of claim 1, wherein the processor is further configured to:

receive from the patient-end digital device, demographic data and biographic data of the patient;

provide the demographic data and biographic data, to a second trained neural network model configured to predict risks for the patient;

receive the predicted risk for the patient from the second trained neural network model; and

provide the predicted risk for the patient as at least one of the one or more physiological inference to the doctor-end digital device.

7. The system of claim 6, wherein the demographic data includes at least one of a location of the patient-end digital device, a zip code in which the patient-end digital device is positioned, a state in which the patient-end digital device is positioned, and a country in which the patient-end digital device is positioned.

8. The system of claim 6, where the biographic data includes at least one of an age of the patient, a dietary information of the patient, and a lifestyle information of the patient.

9. The system of claim 8, wherein the lifestyle information of the patient can include data about physical activity of the patient.

10. The system of claim 6, wherein the predicted risk for the patient can include the risk for medical conditions including at least one of Asthma, Allergies, Flu, and Infectious diseases.

11. A computer-implemented method of predicting a physiological parameter value of a patient based on a facial image of the patient, the method including:

receiving the facial image of the patient from a captured video stream including facial and upper body features of the patient;

processing the facial image to generate a frontal facial image;

providing the frontal facial image, comprising facial and upper body features, to a first trained neural network model configured to predict a sentiment of the patient based on the facial and upper body features;

receiving the sentiment of the patient from the first neural network model indicating how the patient is feeling; wherein the first trained neural network model is a regression deep learning convolutional neural network model; and

providing the predicted physiological parameter value including the sentiment of the patient to aid in real time diagnosis of the patient.

12. The method of claim 11, wherein the first trained neural network model has a plurality of input parameters, the input parameters including three color channels corresponding to one or more images.

13. The method of claim 11, wherein the regression deep learning convolutional neural network model is a Network-in-Network neural network model.

14. The method of claim 11, wherein the frontal facial image is provided via three color channels.

15. The method of claim 11, further including:

receiving demographic data and biographic data of the patient;

providing the demographic data and biographic data, to a second trained neural network model configured to predict risks for the patient;

receiving the predicted risks for the patient from the second trained neural network model; and

providing the predicted physiological parameter value including the risks for the patient to aid in real time diagnosis of the patient.

16. The method of claim 15, wherein the demographic data includes at least one of a location, a zip code, a state, and a country.

17. The method of claim 15, where the biographic data includes at least one of an age of the patient, a dietary information of the patient, and a lifestyle information of the patient.

18. The method of claim 17, wherein the lifestyle information of the patient can include data about physical activity of the patient.

19. The method of claim 15, wherein the predicted risk for the patient can include the risk for medical conditions including at least one of Asthma, Allergies, Flu, and Infectious diseases.

20. A non-transitory computer readable medium including a set of instructions that are executable by one or more processors of a computer to cause the computer to perform a method for predicting a physiological parameter value of a patient based on a facial image of the patient, the method including:

receiving the facial image of the patient from a captured video stream including facial and upper body features of the patient;

processing the facial image to generate a frontal facial image;

providing the frontal facial image, comprising facial and upper body features, to a trained neural network model configured to predict a sentiment of the patient based on the facial and upper body features;

receiving the sentiment of the patient from the trained neural network model indicating how the patient is feeling; wherein the trained neural network model is a regression deep learning convolutional neural network model; and

providing the predicted physiological parameter value including the sentiment of the patient to aid in real time diagnosis of the patient.