Abstract: A video is segmented into a plurality of sequences corresponding to different facial states performed by a patient in the video. For each sequence, displacement of a plurality of groups of landmarks of a face of the patient is tracked, to obtain, for each group of the plurality of groups, one or more displacement measures characterizing positions of the landmarks of the group. The one or more displacement measures corresponding to each group are provided into a corresponding neural network, to obtain a landmark feature. The neural networks corresponding to each group are different from one another. A sequence score for the sequence is determined based on a plurality of landmark features corresponding to the groups. A plurality of sequence scores are provided into a machine learning component, to obtain a patient score. A disease state of the patient is determined based on the patient score.

Abstract: Systems and methods for determining a heath status of a patient through an automated interview. One of the systems include one or more computers in one or more locations and one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations including: providing, to a user interface of a user device, questions for a user to respond to in an interactive manner, in which each of the questions following the first questions is adaptive based on the user's response to one or more of the previous questions; capturing motion and appearance of the user in a video sequence while the user is responding to the questions; and analyzing the motion of the user in the video sequence to determine one or more indications of a disease or of a change in a disease progression.

Abstract: A system and method for monitoring the state of an individual. The method includes providing a stimulus to the individual, measuring a response to the provided stimulus, comparing the measured response to an expected response, and diagnosing one or more aspects of disease in accordance with the result of the comparison between the measured response and the expected response. The stimulus may be a predetermined test sequence, such as a visually displayed predetermined sequence of images, or may include observation of the physical response of the individual while performing one or more predetermined activities. Stored images or video of the individual responding to one or more test sequences may be stored in a lossy or lossless state, and thus security and de-identification may be provided to stored data. This stored data may also be de-identified in a manner to allow for the answering of the greatest number of future questions.

Abstract: A medication confirmation method and apparatus. The method of an embodiment of the invention includes the steps of capturing one or more video sequences of a user administering medication via a medication administration apparatus, storing the captured one or more video sequences, and analyzing at least one of the stored video sequences to confirm that the user has properly administered the medication, and has properly positioned the medication administration apparatus.

Abstract: A method includes: obtaining, by a computer, subject response data including audio data, video data, or audio-video data of a subject's response to one or more stimuli presented to the subject over a defined time period; extracting at least one subject descriptor from the subject response data as a function of time, the at least one subject descriptor including data characterizing involuntary or voluntary action of the subject in response to the one or more stimuli; deriving, from the at least one subject descriptor, a first biomarker characterizing a behavior of the subject in response to the one or more stimuli, in which a value of the first biomarker is indicative of a severity of a disease in the subject; and outputting a disease severity level for the disease as a function of the value of the first biomarker.

Abstract: A system and method for training a system for monitoring administration of medication. The method includes the steps of a method for training a medication administration monitoring apparatus, comprising the steps of defining one or more predetermined medications and then acquiring information from one or more data sources of a user administering medication. A first network is trained to recognize a first step of a medication administration sequence, and then a second network is trained to recognize a second step of a medication administration sequence based upon the training of the first network.

Abstract: A method includes providing one or more instructions to perform a first action associated with administration of a medication; obtaining first data representing a capture of a patient performing the first action based on the one or more instructions; extracting information about movements of the patient from the first data; presenting an image configured to induce an emotional response in the patient; obtaining second data representing capture of the patient performing one or more microexpressions within 250 milliseconds of presentation of the image; obtaining additional data including one or more of demographic information of the patient, information of a disease associated with the patient, or one or more characteristics of the medication; and based on the extracted information about the movements, the one or more microexpressions, and the additional data, determining a medical outcome including a progression of the disease in the patient.

Abstract: A system and method for predictively following up with a user to improve medication adherence. The system includes a medication adherence monitoring apparatus for determining whether a user has taken a medication at a predetermined medication administration time, and a processor for categorizing each determination of whether a user has taken the medication at a predetermined time across a plurality of different dimensions, combining the plurality of different dimensions in a plurality of different combinations to generate a patient adherence score across each of the plurality of different combinations, and ranking a user in accordance with each of the plurality of different combinations. The system further includes a communication apparatus for contacting a user to encourage medication adherence in accordance with at least the ranking of the user in accordance with one or more of the plurality of different combinations.

Abstract: A computer-implemented method includes obtaining a video of a subject, the video including a plurality of frames; generating, based on the plurality of frames, a plurality of optical flows; and encoding the plurality of optical flows using an autoencoder to obtain a movement-based biomarker value of the subject.

Abstract: A system and method for recognizing an object. The system includes an imaging apparatus for capturing an image of an object and a processor for receiving the captured image of the object, and for, when it is determined that fewer than a predetermined number of objects have been previously imaged, for determining whether the image of the captured object includes one or more characteristics determined to be similar to a same characteristic in a group of previously imaged objects so that the captured image is grouped with the previously imaged objects, or whether the image of the captured object includes one or more characteristics determined to be dissimilar to a same characteristic in a group of previously imaged objects so that the captured image is not grouped with the previously imaged objects, and the image of the captured object starts another group of previously imaged objects.

Abstract: A medication management system is described that is operable to determine whether a user is actually following a protocol, provide additional assistance to a user, starting with instructions, video instructions, and the like, and moving up to contact from a medication administrator if it is determined that the user would need such assistance in any medical adherence situation, including clinical trial settings, home care settings, healthcare administration locations, such as nursing homes, clinics, hospitals and the like. Suspicious activity on the part of a patient or other user of the system is identified and can be noted to a healthcare provider or other service provider where appropriate.

Abstract: A method, apparatus and computer program stored on a non-volatile computer readable storage medium for confirming a pill in the mouth of a user. The computer program causing a general purpose computer to perform the steps of capturing one or more images of a user by an image capture device, confirming the position of the face of the user within the captured image by measuring a size of the face, and setting a predetermined portion of the face of the user to be a region of interest. An open mouth of the user is confirmed within the region of interest, and the open mouth of the user is classified as one of a mouth with a pill therein and a mouth without a pill therein.

Abstract: A method, system and computer program for confirming proper medication amount for a medication injection apparatus. The computer program causes a computer to perform the steps of providing one or more instructions on a display of a computing device instructing placement of an injectable medication apparatus including an indication of an amount of medication to be injected by the injectable medication apparatus, imaging by an image capture one or more images of the injectable medication apparatus including the indication of the amount of medication, and comparing by a processor of the computing device through recognition of the indication of the amount of medication, the indicated amount to a predetermined amount. One or more additional instructions may be provided on the display of the computing device to change the indication of the amount of medication if it is determined that the recognized indicated amount does not match the predetermined amount.

Abstract: Methods and systems for recognizing suspicious activities of a user administering a medication are described. One of the methods include capturing one or more video sequences of a user administering a medication by a video capture device; analyzing the captured one or more video sequences to determine one or more indications of a suspicious activity of the user; and adaptively performing a follow-up operation in response to a determination of one or more indications of a suspicious activity of the user.

Abstract: A medication confirmation method and apparatus for confirming administration of medication employing an inhalable medication administration apparatus. The method of an embodiment of the invention includes the steps of capturing one or more video sequences of a user administering medication employing the inhalable medication administration apparatus, storing the captured one or more video sequences, capturing one or more audio sequences of the user administering medication employing the inhalable medication administration apparatus and storing the captured one or more audio sequences. At least one of the stored video sequences and at least one of the stored audio sequences are then analyzed to confirm that the user has properly administered the medication.

Abstract: A method and system for de-identifying a video sequence are provided. The method may include the steps of capturing a video sequence, comprising a number of individual frames, including one or more users performing one or more actions, and using activity recognition to recognize one of the one or more actions. One or more of the plurality of frames may be defined as comprising the recognized one or more actions, and a portion of the one or more of the plurality of frames may be identified to remain visible. The non-identified portions of the one or more of the plurality of frames and the non-defined frames may be de-identified. This method may be applied to the determination of whether a user has ingested a medication pill.

Abstract: Systems and methods for determining a heath status of a patient through an automated interview. One of the systems include one or more computers in one or more locations and one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations including: providing, to a user interface of a user device, questions for a user to respond to in an interactive manner, in which each of the questions following the first questions is adaptive based on the user's response to one or more of the previous questions; capturing motion and appearance of the user in a video sequence while the user is responding to the questions; and analyzing the motion of the user in the video sequence to determine one or more indications of a disease or of a change in a disease progression.

Abstract: A medication management system is described that is operable to determine whether a user is actually following a protocol, provide additional assistance to a user, starting with instructions, video instructions, and the like, and moving up to contact from a medication administrator if it is determined that the user would need such assistance in any medical adherence situation, including clinical trial settings, home care settings, healthcare administration locations, such as nursing homes, clinics, hospitals and the like. Suspicious activity on the part of a patient or other user of the system is identified and can be noted to a healthcare provider or other service provider where appropriate.

Abstract: A system and method for monitoring the state of an individual. The method includes providing a stimulus to the individual, measuring a response to the provided stimulus, comparing the measured response to an expected response, and diagnosing one or more aspects of disease in accordance with the result of the comparison between the measured response and the expected response. The stimulus may be a predetermined test sequence, such as a visually displayed predetermined sequence of images, or may include observation of the physical response of the individual while performing one or more predetermined activities. Stored images or video of the individual responding to one or more test sequences may be stored in a lossy or lossless state, and thus security and de-identification may be provided to stored data. This stored data may also be de-identified in a manner to allow for the answering of the greatest number of future questions.

Abstract: A system and method for recognizing an object. The system includes an imaging apparatus for capturing an image of an object and a processor for receiving the captured image of the object, and for, when it is determined that fewer than a predetermined number of objects have been previously imaged, for determining whether the image of the captured object includes one or more characteristics determined to be similar to a same characteristic in a group of previously imaged objects so that the captured image is grouped with the previously imaged objects, or whether the image of the captured object includes one or more characteristics determined to be dissimilar to a same characteristic in a group of previously imaged objects so that the captured image is not grouped with the previously imaged objects, and the image of the captured object starts another group of previously imaged objects.