Abstract: Various of the disclosed embodiments provide Graphical User Interfaces (GUIs) for reviewing prior surgical procedures. Specifically, the GUI may allow a user, such as surgeon, to review sensor data, including video data, acquired during various of the surgeon's past surgical procedures. Some sensor data may be organized into metrics referred to herein as objective performance indicators (OPIs). Similarly, procedures may be discretized into specific tasks. By organizing and presenting data in OPI form at the task level, the GUI may facilitate efficient and coordinated review of the surgeon's progress over time across multiple procedures. In some embodiments, corresponding data from expert surgeons may also presented in the interface so that the user may gauge the surgeon's relative performance.

Abstract: Various of the disclosed embodiments relate to systems and methods for recognizing types of surgical operations from data gathered in a surgical theater, such as recognizing a surgery procedure and corresponding specialty from endoscopic video data. Some embodiments select discrete frame sets from the data for individual consideration by a corpus of machine learning models, Some embodiments may include an uncertainty indication with each classification to guide downstream decision-making based upon the classification. For example, where the system is used as part of a data annotation pipeline, uncertain classifications may be flagged for downstream confirmation and review by a human reviewer.

Abstract: Various of the disclosed embodiments are directed to computer-implemented systems and methods for recognizing surgical tasks from surgical data, In some embodiments an ensemble model configured to receive video data, kinematics data, and system event data from the surgical theater may be implemented. The ensemble model may implement modular streams for processing the data, facilitating predictions even when less than all the data types are available. In some embodiments, smoothing operations may help facilitate more accurate prediction results, Various of the embodiments may be employed in real-time during surgery, providing predictions at per-second intervals.

Abstract: Various of the disclosed embodiments are directed to systems and computer-implemented methods for determining surgical system events and/or kinematic data based upon surgical video data, such as video acquired at an endoscope. In some embodiments, derived data may be inferred from elements appearing in a graphical user interface (GUI) exclusively. Icons and text may be recognized in the GUI to infer event occurrences and tool actions. In some embodiments, derived data may additionally, or alternatively, be inferred from optical flow values derived from the video and by tracking tools entering and leaving the video field of view. Some embodiments include logic for reconciling data values derived from each of these approaches.

Abstract: Various of the disclosed embodiments relate to systems and methods for processing surgical data to facilitate further downstream operations. For example, some embodiments may include machine learning systems trained to recognize whether video from surgical visualization tools, such as endoscopes, depicts a field of view inside or outside the patient body. The system may excise or whiteout frames of video appearing outside the patient so as to remove potentially compromising personal information, such as the identities of members of the surgical team, the patients identity, configurations of the surgical theater, etc. Appropriate removal of such non-surgical data may facilitate downstream processing, e.g., by complying with regulatory requirements as well as by removing extraneous data potentially inimical to further downstream processing, such as training a downstream classifier.