Abstract: A method for predicting movement of a first plurality of keypoints of a first instrument comprises receiving, at a neural network model, a first location of the first plurality of keypoints and a first location of a second plurality of keypoints of a second instrument. The method further comprises determining a trajectory for the first and second pluralities of keypoints by: generating, using an attention model of the neural network model, a first and second tool-level graph indicating a spatial-temporal relationship between the first and second pluralities of keypoints, respectively; and generating a scene-level graph based on the tool-level graphs. The scene-level graph indicates a spatial-temporal relationship between the first and second pluralities of keypoints. The method further comprises generating an output image based on the determined trajectory. The output image includes an output location of the first and second pluralities of keypoints.

Abstract: One embodiment of the present invention sets forth a technique for training a machine learning model to perform feature extraction. The technique includes executing a student version of the machine learning model to generate a first set of features from a first set of image crops and executing a teacher version of the machine learning model to generate a second set of features from a second set of image crops. The technique also includes training the student version of the machine learning model based on one or more losses computed between the first and second sets of features. The technique further includes transmitting the trained student version of the machine learning model to a server, wherein the trained student version can be aggregated by the server with additional trained student versions of the machine learning model to generate a global version of the machine learning model.

Abstract: A method for detecting a location of a plurality of keypoints of a surgical instrument comprises receiving, at a first neural network model, a video input of a surgical procedure. The method further comprises generating, using the first neural network model, a first output image including a first output location of the plurality of keypoints annotated on a first output image of the surgical instrument. The method further comprises receiving, at a second neural network model, the first output image and historic keypoint trajectory data including a historic trajectory for the plurality of keypoints. The method further comprises determining, using the second neural network model, a trajectory for the plurality of keypoints. The method further comprises generating, using the second neural network model, a second output image including a second output location of the plurality of keypoints annotated on a second output image of the surgical instrument.

Abstract: Various of the disclosed embodiments relate to systems and methods for recognizing types of surgical operations from data gathered in a surgical theater, such as recognizing a surgery procedure and corresponding specialty from endoscopic video data. Some embodiments select discrete frame sets from the data for individual consideration by a corpus of machine learning models, Some embodiments may include an uncertainty indication with each classification to guide downstream decision-making based upon the classification. For example, where the system is used as part of a data annotation pipeline, uncertain classifications may be flagged for downstream confirmation and review by a human reviewer.

Abstract: Various of the disclosed embodiments relate to systems and methods for processing surgical data to facilitate further downstream operations. For example, some embodiments may include machine learning systems trained to recognize whether video from surgical visualization tools, such as endoscopes, depicts a field of view inside or outside the patient body. The system may excise or whiteout frames of video appearing outside the patient so as to remove potentially compromising personal information, such as the identities of members of the surgical team, the patients identity, configurations of the surgical theater, etc. Appropriate removal of such non-surgical data may facilitate downstream processing, e.g., by complying with regulatory requirements as well as by removing extraneous data potentially inimical to further downstream processing, such as training a downstream classifier.

Abstract: Systems, methods, and non-transitory computer-readable media can collect a set of training videos as training data, wherein the set of training videos are labeled with one or more labels based on one or more video quality metrics associated with an evaluation objective. A machine learning model is trained based on the training data. A video to be evaluated is received. The video is assigned to a first video quality category of a plurality of video quality categories based on the machine learning model.

Abstract: Systems, methods, and non-transitory computer-readable media can collect a set of training videos as training data, wherein the set of training videos are labeled with one or more labels based on one or more video quality metrics associated with an evaluation objective. A machine learning model is trained based on the training data. A video to be evaluated is received. The video is assigned to a first video quality category of a plurality of video quality categories based on the machine learning model.

Abstract: According to an aspect of the invention, there is provided a low back pain analysis device comprising: a processor; and a storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by the processor, perform processing of: obtaining a relationship between a result of a pattern and a low back pain, using the result of the pattern obtained by classifying gravity center movement data acquired by a sensor, which is attached to furniture and acquires the gravity center movement data for a sitting period including a period for which a person is sitting on the furniture, using clustering.

Abstract: Systems, methods, and non-transitory computer-readable media can receive a set of video frames associated with a video. For each video frame of the set of video frames, a plurality of interest points are identified based on an interest point detector. For each video frame of the set of video frames, it is determined whether the video frame depicts the same static image as a next video frame in the set of video frames based on the plurality of interest points identified in each video frame.

Abstract: Systems, methods, and non-transitory computer-readable media can receive a set of video frames associated with a video. Dynamic regions in each video frame of the set of video frames can be filtered out, wherein each dynamic region represents a region in which a threshold level of movement is detected. A determination can be made for each video frame of the set of filtered video frames, whether the video frame comprises synthetic overlaid text based on a machine learning model.

Abstract: Systems, methods, and non-transitory computer-readable media can receive a set of video frames associated with a video. A determination can be made that a first set of consecutive video frames of the set of video frames depicts identical content to a second set of consecutive video frames of the set of video frames, wherein the first set of consecutive video frames and the second set of consecutive video frames satisfy a threshold number of consecutive video frames. The video is identified as a looping video based on the determination that the first set of consecutive video frames depicts identical content to the second set of consecutive video frames.

Abstract: Systems, methods, and non-transitory computer-readable media can receive a set of video frames associated with a video. A determination can be made that a threshold number of video frames of the set of video frames depict two or more reaction icons of a set of reaction icons. The video can be identified as a poll video based on the determining that the threshold number of video frames of the set of video frames depict two or more reaction icons of the set of reaction icons.

Abstract: Systems, methods, and non-transitory computer-readable media can collect a set of training videos as training data, wherein the set of training videos are labeled with one or more labels based on one or more video quality metrics associated with an evaluation objective. A machine learning model is trained based on the training data. A video to be evaluated is received. The video is assigned to a first video quality category of a plurality of video quality categories based on the machine learning model.

Abstract: A virtual reality system includes a head-mounted display (HMD) having one or more facial sensors and illumination sources mounted to a surface of the HMD. For example, the facial sensors are image capture devices coupled to a bottom side of the HMD. The illumination sources illuminate portions of a user's face outside of the HMD, while the facial sensors capture images of the illuminated portions of the user's face. A controller receives the captured images and generates a representation of the portions of the user's face by identifying landmarks of the user's face in the captured images and performing other suitable image processing methods. Based on the representation, the controller or another component of the virtual reality system generates content for presentation to the user.

Abstract: A facial tracking system generates a virtual rendering of a portion of a face of a user wearing a head-mounted display (HMD). The facial tracking system illuminates portions of the face inside the HMD. The facial tracking system captures a plurality of facial data of the portion of the face using one or more facial sensors located inside the HMD. A plurality of planar sections of the portion of the face are identified based at least in part on the plurality of facial data. The plurality of planar sections are mapped to one or more landmarks of the face. Facial animation information is generated based at least in part on the mapping, the facial animation information describing a portion of a virtual face corresponding to the portion of the user's face.

Abstract: A head mounted display (HMD) in a VR system includes sensors for tracking the eyes and face of a user wearing the HMD. The VR system records calibration attributes such as landmarks of the face of the user. Light sources illuminate portions of the user's face covered by the HMD. In conjunction, facial sensors capture facial data. The VR system analyzes the facial data to determine the orientation of planar sections of the illuminated portions of face. The VR system aggregates planar sections of the face and maps the planar sections to landmarks of the face to generate a facial animation of the user, which can also include eye orientation information. The facial animation is represented as a virtual avatar and presented to the user.

Abstract: A head mounted display (HMD) in a VR system includes sensors for tracking the eyes and face of a user wearing the HMD. The VR system records calibration attributes such as landmarks of the face of the user. Light sources illuminate portions of the user's face covered by the HMD. In conjunction, facial sensors capture facial data. The VR system analyzes the facial data to determine the orientation of planar sections of the illuminated portions of face. The VR system aggregates planar sections of the face and maps the planar sections to landmarks of the face to generate a facial animation of the user, which can also include eye orientation information. The facial animation is represented as a virtual avatar and presented to the user.

Abstract: A facial tracking system generates a virtual rendering of a portion of a face of a user wearing a head-mounted display (HMD). The facial tracking system illuminates portions of the face inside the HMD. The facial tracking system captures a plurality of facial data of the portion of the face using one or more facial sensors located inside the HMD. A plurality of planar sections of the portion of the face are identified based at least in part on the plurality of facial data. The plurality of planar sections are mapped to one or more landmarks of the face. Facial animation information is generated based at least in part on the mapping, the facial animation information describing a portion of a virtual face corresponding to the portion of the user's face.

Abstract: A virtual reality system includes a head-mounted display (HMD) having one or more facial sensors and illumination sources mounted to a surface of the HMD. For example, the facial sensors are image capture devices coupled to a bottom side of the HMD. The illumination sources illuminate portions of a user's face outside of the HMD, while the facial sensors capture images of the illuminated portions of the user's face. A controller receives the captured images and generates a representation of the portions of the user's face by identifying landmarks of the user's face in the captured images and performing other suitable image processing methods. Based on the representation, the controller or another component of the virtual reality system generates content for presentation to the user.