Abstract: Systems and methods for depth detection and virtual modeling using surgical robots during a surgical procedure are described. The robotic surgical systems include robotic arms with interchangeable surgical tools. An endoscope at the end of one of the robotic arms includes a depth sensor for detecting a distance from the camera to patient anatomy. The depth data and image data are used to generate a feature model of the patient anatomy for reference by a surgeon. The feature model is displayed to the surgeon in an augmented reality view with preoperative and intraoperative images mapped thereon for reference and guidance of a surgical procedure.

Abstract: Examples of systems and methods for detecting delays during a surgical procedure are disclosed. For example, one disclosed method includes receiving, a computing device, video of a robotic surgical procedure; determining, by the computing device, an estimated time-to-complete (“TTC”) the robotic surgical procedure based on the video; and in response to determining a deviation between the estimated TTC and an expected duration of the robotic surgical procedure exceeds a threshold, outputting an indication that the robotic surgical procedure is deviating from the expected duration.

Abstract: Systems and methods for identifying surgery tools of a robotic surgery system in surgery video based on robot kinematics are described. Kinematic data describing a position of a first robotic surgery arm including a surgery tool is used to determine a nominal position of the first robotic surgery arm. A search area within image data obtained during a robotic surgery from an endoscope attached to a second robotic surgery arm is determined based on the kinematic data and a nominal position of the surgery tool. The surgery tool is identified within the search area of the image data using an object detection technique.

Abstract: One example method for detecting phases of a surgical procedure via video processing includes accessing a video of the surgical procedure and dividing the video into one or more blocks, each of the blocks containing one or more video frames. For each of the blocks, the method includes applying a prediction model on the video frames of the respective block to obtain a phase prediction for each of the video frames. The prediction model is configured to predict, for an input video frame, one of the plurality of phases of the surgical procedure. The method further includes generating an aggregated phase prediction for the respective block by aggregating the phase predictions of the video frames, and modifying the video of the surgical procedure to include an indication of a predicted phase of the respective block based on the aggregated phase prediction.

Abstract: One example method for robotic surgery using multi-user authentication without credentials includes receiving, by a robotic surgical device, a case code associated with a medical procedure; validating the case code; determining one or more users associated with the case code; and providing access to the one or more users to the robotic surgical device to enable a robotic surgical procedure.

Abstract: One example method for identifying useful segments in surgical videos includes accessing a video of a surgical procedure and user activities of a plurality of users who have watched the video of the surgical procedure. The user activities include operations performed during playback of the video. The method further includes dividing the video into multiple segments and determining a popularity score for each of the multiple segments based on the operations. Useful segments are identified from the segments based on the popularity scores. The method further includes generating metadata for the video of the surgical procedure to include an indication of the identified useful segments and associating the metadata with the video of the surgical procedure.

Abstract: One example method for detecting phases of a surgical procedure via video processing includes accessing a video of the surgical procedure and dividing the video into one or more blocks, each of the blocks containing one or more video frames. For each of the blocks, the method includes applying a prediction model on the video frames of the respective block to obtain a phase prediction for each of the video frames. The prediction model is configured to predict, for an input video frame, one of the plurality of phases of the surgical procedure. The method further includes generating an aggregated phase prediction for the respective block by aggregating the phase predictions of the video frames, and modifying the video of the surgical procedure to include an indication of a predicted phase of the respective block based on the aggregated phase prediction.

Abstract: One method for fleet management of robotic surgical systems includes receiving, by a management server from a robotic surgery system, a provisioning request; in response to receiving the provisioning request: generating an encryption key pair for the robotic surgery system, the encryption key pair comprising a private key and a public key, communicating the private key to the robotic surgery system, and communicating a set of secure certificates to the robotic surgery system, at least one of the secure certificates enabling secure communications between the robotic surgery system and the management server; receiving from the robotic surgery system, and using the at least one secure certificate enabling secure communications, a message indicating one or more software packages, each software package indicating a version of an installed software package on the robotic surgery system; communicating one or more software updates to the robotic surgery system based on the message; and registering, at the management

Abstract: Systems and methods for surgical training are provided. The system may include an operating display console for a first surgeon to view a video feed of a surgery while the first surgeon performs the surgery. The system may also include a training display console for a second surgeon to view the video feed of the surgery while the surgery is performed. The system may further include a controller that interfaces with the operating display console and the training display console. The training display console may include a training interface for pausing the video feed displayed on the training display console, and for switching from a display mode to a simulation mode for simulating one or more surgical steps of the surgery.

Abstract: One example method for detecting phases of a surgical procedure via video processing includes accessing a video of the surgical procedure and dividing the video into one or more blocks, each of the blocks containing one or more video frames. For each of the blocks, the method includes applying a prediction model on the video frames of the respective block to obtain a phase prediction for each of the video frames. The prediction model is configured to predict, for an input video frame, one of the plurality of phases of the surgical procedure. The method further includes generating an aggregated phase prediction for the respective block by aggregating the phase predictions of the video frames, and modifying the video of the surgical procedure to include an indication of a predicted phase of the respective block based on the aggregated phase prediction.

Abstract: One example method for improving robotic surgical safety via video processing includes identifying, during a robotic surgical procedure, one or more surgical tools using one or more images of the surgical procedure captured by a camera, the robotic surgical procedure employing a robotic surgical device controlling the one or more identified surgical tools; predicting, for at least one of the one or more images, one or more loaded surgical tools that are controlled by the robotic surgical device and should be in a field of view of the camera; comparing the one or more identified surgical tools with the one or more loaded surgical tools; determining that at least one of the one or more loaded surgical tools does not match any of the one or more identified surgical tools; and causing the at least one loaded tool of the robotic surgical device to be disabled.

Abstract: A system embodied in an article of clothing for synchronizing one or more breast-pumping sessions of an individual (such as a mother) and milk consumption by a second individual (such as a baby or infant) is described. In particular, based on measurements of a volume of the collected milk as a function of time and received information specifying milk consumption by the second individual as a function of time, a control circuit may determine a need for milk. Then, the control circuit may provide feedback based on the determined need for milk that synchronizes the one or more breast-pumping sessions and the milk consumption. For example, the feedback may alert the individual to initiate a breast-pumping session and/or may provide a signal to a breast pump that initiates a breast-pumping session.

Abstract: An apparatus, system and process for guiding a surgeon during a medical procedure to prevent surgical mistakes are described. The system may include a machine learning medical procedure server that generates one or more machine learning medical procedure models using, at least, medical procedure data captured during medical procedures performed at a plurality of different medical procedure systems. The system may also include a medical procedure system communicably coupled with the machine learning medical procedure server that receives a selected machine learning medical procedure model from the machine learning medical procedure server, and utilizes the selected machine learning medical procedure model during a corresponding medical procedure to control one or more operations of the medical procedure system.

Abstract: Systems and methods for surgical training are provided. The system may include an operating display console for a first surgeon to view a video feed of a surgery while the first surgeon performs the surgery. The system may also include a training display console for a second surgeon to view the video feed of the surgery while the surgery is performed. The system may further include a controller that interfaces with the operating display console and the training display console. The training display console may include a training interface for pausing the video feed displayed on the training display console, and for switching from a display mode to a simulation mode for simulating one or more surgical steps of the surgery.

Abstract: One example method for improving robotic surgical safety via video processing includes identifying, during a robotic surgical procedure, one or more surgical tools using one or more images of the surgical procedure captured by a camera, the robotic surgical procedure employing a robotic surgical device controlling the one or more identified surgical tools; predicting, for at least one of the one or more images, one or more loaded surgical tools that are controlled by the robotic surgical device and should be in a field of view of the camera; comparing the one or more identified surgical tools with the one or more loaded surgical tools; determining that at least one of the one or more loaded surgical tools does not match any of the one or more identified surgical tools; and causing the at least one loaded tool of the robotic surgical device to be disabled.

Abstract: A system for synchronizing one or more breast-pumping sessions of an individual (such as a mother) and milk consumption by a second individual (such as a baby or infant) is described. In particular, based on measurements of a volume of the collected milk as a function of time and received information specifying milk consumption by the second individual as a function of time, a control circuit may determine a need for milk. Then, the control circuit may provide feedback based on the determined need for milk that synchronizes the one or more breast-pumping sessions and the milk consumption. For example, the feedback may alert the individual to initiate a breast-pumping session and/or may provide a signal to a breast pump that initiates a breast-pumping session.

Abstract: One example method for identifying useful segments in surgical videos includes accessing a video of a surgical procedure and user activities of a plurality of users who have watched the video of the surgical procedure. The user activities include operations performed during playback of the video. The method further includes dividing the video into multiple segments and determining a popularity score for each of the multiple segments based on the operations. Useful segments are identified from the segments based on the popularity scores. The method further includes generating metadata for the video of the surgical procedure to include an indication of the identified useful segments and associating the metadata with the video of the surgical procedure.

Abstract: In some embodiments, a surgery assistance system is provided. The surgery assistance system comprises an image sensor, a video capture computing device, a notification computing device, and a machine learning (ML) processing computing device. The ML processing computing device is configured to receive video data, generate copies of the video data downsampled as appropriate for each of a plurality of machine learning models, process the copies of the video data using the machine learning models, and cause the notification computing device to provide at least one notification based on the machine learning models detecting at least one instance of an item.

Abstract: An apparatus, system and process for guiding a surgeon during a medical procedure to prevent surgical mistakes are described. The system may include a machine learning medical procedure server that generates one or more machine learning medical procedure models using, at least, medical procedure data captured during medical procedures performed at a plurality of different medical procedure systems. The system may also include a medical procedure system communicably coupled with the machine learning medical procedure server that receives a selected machine learning medical procedure model from the machine learning medical procedure server, and utilizes the selected machine learning medical procedure model during a corresponding medical procedure to control one or more operations of the medical procedure system.

Abstract: One example method for identifying useful segments in surgical videos includes accessing a video of a surgical procedure and user activities of a plurality of users who have watched the video of the surgical procedure. The user activities include operations performed during playback of the video. The method further includes dividing the video into multiple segments and determining a popularity score for each of the multiple segments based on the operations. Useful segments are identified from the segments based on the popularity scores. The method further includes generating metadata for the video of the surgical procedure to include an indication of the identified useful segments and associating the metadata with the video of the surgical procedure.