Abstract: Systems and methods for depth detection and virtual modeling using surgical robots during a surgical procedure are described. The robotic surgical systems include robotic arms with interchangeable surgical tools. An endoscope at the end of one of the robotic arms includes a depth sensor for detecting a distance from the camera to patient anatomy. The depth data and image data are used to generate a feature model of the patient anatomy for reference by a surgeon. The feature model is displayed to the surgeon in an augmented reality view with preoperative and intraoperative images mapped thereon for reference and guidance of a surgical procedure.

Abstract: Examples of systems and methods for detecting delays during a surgical procedure are disclosed. For example, one disclosed method includes receiving, a computing device, video of a robotic surgical procedure; determining, by the computing device, an estimated time-to-complete (“TTC”) the robotic surgical procedure based on the video; and in response to determining a deviation between the estimated TTC and an expected duration of the robotic surgical procedure exceeds a threshold, outputting an indication that the robotic surgical procedure is deviating from the expected duration.

Abstract: Systems and methods for identifying surgery tools of a robotic surgery system in surgery video based on robot kinematics are described. Kinematic data describing a position of a first robotic surgery arm including a surgery tool is used to determine a nominal position of the first robotic surgery arm. A search area within image data obtained during a robotic surgery from an endoscope attached to a second robotic surgery arm is determined based on the kinematic data and a nominal position of the surgery tool. The surgery tool is identified within the search area of the image data using an object detection technique.

Abstract: One example method for improving the efficiency of robotic surgical procedures via surgical procedure data analysis. The method includes accessing surgical procedure data of a robotic surgical procedure. The surgical procedure data contains data or events associated with the robotic surgical system during the robotic surgical procedure. The method further includes accessing a procedure setup plan for the robotic surgical procedure corresponding to the surgical procedure data and determining an idle period of the surgical procedure based on the surgical procedure data. The method also includes analyzing the surgical procedure data to detect one or more events associated with the idle period to determine a cause of the idle period, and generating a recommendation for modifying the procedure setup plan for the robotic surgical procedure based on the determined cause of the idle period.

Abstract: One example method for improving robotic surgical safety via video processing includes identifying, during a robotic surgical procedure, one or more surgical tools using one or more images of the surgical procedure captured by a camera, the robotic surgical procedure employing a robotic surgical device controlling the one or more identified surgical tools; predicting, for at least one of the one or more images, one or more loaded surgical tools that are controlled by the robotic surgical device and should be in a field of view of the camera; comparing the one or more identified surgical tools with the one or more loaded surgical tools; determining that at least one of the one or more loaded surgical tools does not match any of the one or more identified surgical tools; and causing the at least one loaded tool of the robotic surgical device to be disabled.

Abstract: One example method for improving robotic surgical safety via video processing includes identifying, during a robotic surgical procedure, one or more surgical tools using one or more images of the surgical procedure captured by a camera, the robotic surgical procedure employing a robotic surgical device controlling the one or more identified surgical tools; predicting, for at least one of the one or more images, one or more loaded surgical tools that are controlled by the robotic surgical device and should be in a field of view of the camera; comparing the one or more identified surgical tools with the one or more loaded surgical tools; determining that at least one of the one or more loaded surgical tools does not match any of the one or more identified surgical tools; and causing the at least one loaded tool of the robotic surgical device to be disabled.

Abstract: A user interface system includes one or more controllers configured to move freely in three dimensions, where the one or more controllers include an inertial sensor coupled to measure movement of the one or more controllers, and output movement data including information about the movement. The user interface system further includes a processor coupled to receive the movement data, where the processor includes logic that, when executed by the processor, causes the user interface system to perform operations, including receiving the movement data with the processor, identifying an unintentional movement in the movement data with the processor, and outputting unintentional movement data that identifies the unintentional movement.

Abstract: One example method for detecting events during a surgery includes receiving surgical video of a surgery comprising a plurality of video frames; identifying, by a first trained machine-learning (“ML”) model, an event during a surgical procedure based on the surgical video; determining, by the first trained ML model, a subset of the plurality of video frames, the subset of the plurality of video frames corresponding to the event; determining, by a second trained ML model, a characteristic of the event based on the subset of the plurality of video frames; and generating metadata corresponding to event based on the characteristic of the event.

Abstract: Systems and methods for depth detection and virtual modeling using surgical robots during a surgical procedure are described. The robotic surgical systems include robotic arms with interchangeable surgical tools. An endoscope at the end of one of the robotic arms includes a depth sensor for detecting a distance from the camera to patient anatomy. The depth data and image data are used to generate a feature model of the patient anatomy for reference by a surgeon. The feature model is displayed to the surgeon in an augmented reality view with preoperative and intraoperative images mapped thereon for reference and guidance of a surgical procedure.

Abstract: A method includes receiving a first set of image data representing passageways at a first cyclical motion state, receiving a second set of image data representing the passageways at a second cyclical motion state, receiving pose data for points describing a shape of an instrument, and comparing the shape of the instrument to the first and second sets of image data. The comparing includes assigning match scores to the sets of image data by comparing each to the shape of the instrument and determining a selected set of image data that matches the shape. The method further includes identifying a phase of a cyclical anatomical motion, generating a command signal indicating an intended movement of the instrument, adjusting the command signal to include an instruction for a cyclical instrument motion of the instrument based on the phase of the cyclical anatomical motion, and causing the intended movement of the instrument.

Abstract: Systems and methods for identifying surgery tools of a robotic surgery system in surgery video based on robot kinematics are described. Kinematic data describing a position of a first robotic surgery arm including a surgery tool is used to determine a nominal position of the first robotic surgery arm. A search area within image data obtained during a robotic surgery from an endoscope attached to a second robotic surgery arm is determined based on the kinematic data and a nominal position of the surgery tool. The surgery tool is identified within the search area of the image data using an object detection technique.

Abstract: Various arrangements for monitoring and controlling operation of a HVAC system are provided. Battery-operated temperature monitoring devices may be situated within the structure and connected with a network. A user command may be received that includes a selection of at least one of the battery-operated temperature monitoring devices. An offset may be calculated based on an evaluation of a first temperature reading of the selected battery-operated temperature monitoring device relative to a second temperature reading of the thermostat. The offset may be overridden when an override condition is present.

Abstract: One example method for improving robotic surgical safety via video processing includes identifying, during a robotic surgical procedure, one or more surgical tools using one or more images of the surgical procedure captured by a camera, the robotic surgical procedure employing a robotic surgical device controlling the one or more identified surgical tools; predicting, for at least one of the one or more images, one or more loaded surgical tools that are controlled by the robotic surgical device and should be in a field of view of the camera; comparing the one or more identified surgical tools with the one or more loaded surgical tools; determining that at least one of the one or more loaded surgical tools does not match any of the one or more identified surgical tools; and causing the at least one loaded tool of the robotic surgical device to be disabled.

Abstract: A user interface system includes one or more controllers configured to move freely in three dimensions, where the one or more controllers include an inertial sensor coupled to measure movement of the one or more controllers, and output movement data including information about the movement. The user interface system further includes a processor coupled to receive the movement data, where the processor includes logic that, when executed by the processor, causes the user interface system to perform operations, including receiving the movement data with the processor, identifying an unintentional movement in the movement data with the processor, and outputting unintentional movement data that identifies the unintentional movement.

Abstract: A method for monitoring and controlling operation of a HVAC system within a structure comprises providing a thermostat configured to regulate the HVAC system. The method includes situating a plurality of battery-operated temperature monitoring devices within the structure and coupling the thermostat and each of the plurality of battery-operated temperature monitoring devices to a network. The method comprises receiving over the network a user command that includes a selection of at least one of the battery-operated temperature monitoring devices. The method includes computing at the thermostat an offset based on an evaluation of a first temperature reading of the at least one selected battery-operated temperature monitoring device relative to a second temperature reading of the thermostat. The method comprises applying the offset to the thermostat to cause the thermostat to regulate the HVAC system in view of the first temperature reading, and overriding the offset when an override condition is met.

Abstract: A method of modeling a cyclic anatomical motion comprises receiving a pose dataset for an identified point on an interventional instrument retained within and in compliant movement with a cyclically moving patient anatomy for a plurality of time parameters. The method also includes determining a set of pose differentials for the identified point with respect to a reference point at each of the plurality of time parameters and identifying a periodic signal for the cyclic anatomical motion from the set of pose differentials.

Abstract: Measured locations of position sensors on a steerable medical device are used to determine the insertion depth of the steerable medical device with respect to a foundation point, which is initialized by an operator. The insertion depth, in turn, is used to determine the state of each segment that has passed the foundation point, i.e., is distal to the foundation point.

Abstract: Measured locations of position sensors on a steerable medical device are used to determine the insertion depth of the steerable medical device with respect to a foundation point, which is initialized by an operator. The insertion depth, in turn, is used to determine the state of each segment that has passed the foundation point, i.e., is distal to the foundation point.