Abstract: A method for detecting procedure milestone events with a spatial context may comprise receiving a video input including a plurality of video frame images of a procedure and analyzing the video input using a neural network model that includes a multi-head attention mechanism. The method may also include generating, with a spatial context head of the multi-head attention mechanism, a spatial context indicator for at least a portion of the plurality of video frame images. The method may also include generating, with a classification head of the multi-head attention mechanism, a milestone indicator for at least one of the plurality of video frame images of the plurality of video frame images and generating a procedure evaluation output that includes the milestone indicator and the spatial context indicator.

Abstract: The solution for an ML-based medical procedure analysis with interpretable model confidence rankings is disclosed. The solution can include a system having one or more processors, coupled with memory. The system can receive a plurality of input features associated with a prediction for a video stream that captures a procedure performed with a robotic medical system. The prediction can be made via a first model trained with machine learning. The system can determine, via a second model trained with machine learning, a level of confidence in the prediction made via the first model. The system can attribute the level of confidence among at least two input features of the plurality of input features. The system can provide, for display via a display device, an indication overlaid on the video stream of the attribution of the level of confidence among the at least two input features.

Abstract: Multi-modal data and ontology knowledge fusion machine learning for robotic medical systems is described. One or more processors can generate, using the training dataset and one or more teacher models, classifications of segments of the medical procedures in a first segment type. The one or more processors can map, using an ontology indicating a hierarchy of different segment types of medical procedures, the classifications of the segments in the first segment type to a second segment type. The one or more processors can train, using the mapping based on the classifications generated by the one or more teacher models, one or more student models with a machine learning technique. The one or more processors can execute, using data received from a robotic medical system for a medical procedure, the one or more student models to classify a segment of the medical procedure.

Abstract: The technical solutions are directed to a multi-modal retrieval augmented generation for natural language interactions with surgical video and data. A system can include a processor coupled with memory. The processor can identify, for a medical procedure performed via a robotic medical system, a video stream and a plurality of data streams related to the medical procedure. The processor can determine, using one or more models trained with machine learning, based on the plurality of data streams, performance data for a clip of the video stream. The processor can transform the performance data and the clip to an embedding vector for an embedding space stored in a data repository. The processor can update the embedding space to provide, in response to a search query executed on the embedding space, access to at least a portion of the video stream of the medical procedure.

Abstract: Extracting features to compress images generated during medical procedures is provided. In examples, systems are configured to obtain one or more frames of a video captured by a camera of a medical procedure performed with a robotic medical system. Systems can be configured to generate features for the one or more frames using a first model trained with self-supervised machine learning and constructing a dataset based on the generated features. Some systems can be configured to construct a dataset based on the generated features and input the dataset into a second model to detect an aspect of the medical procedure.

Abstract: The solution for an ML-based medical procedure analysis with interpretable model confidence rankings is disclosed. The solution can include a system having one or more processors, coupled with memory. The system can receive a plurality of input features associated with a prediction for a video stream that captures a procedure performed with a robotic medical system. The prediction can be made via a first model trained with machine learning. The system can determine, via a second model trained with machine learning, a level of confidence in the prediction made via the first model. The system can attribute the level of confidence among at least two input features of the plurality of input features. The system can provide, for display via a display device, an indication overlaid on the video stream of the attribution of the level of confidence among the at least two input features.

Abstract: One embodiment of the present invention sets forth a technique for training a machine learning model to perform feature extraction. The technique includes executing a student version of the machine learning model to generate a first set of features from a first set of image crops and executing a teacher version of the machine learning model to generate a second set of features from a second set of image crops. The technique also includes training the student version of the machine learning model based on one or more losses computed between the first and second sets of features. The technique further includes transmitting the trained student version of the machine learning model to a server, wherein the trained student version can be aggregated by the server with additional trained student versions of the machine learning model to generate a global version of the machine learning model.