Abstract: Described herein are systems, methods, and instrumentalities associated with automatically annotating a 3D image dataset. The 3D automatic annotation may be accomplished based on a 2D manual annotation provided by an annotator and by propagating, using a set of machine-learning (ML) based techniques, the 2D manual annotation through sequences of 2D images associated with the 3D image dataset. The automatically annotated 3D image dataset may then be used to annotate other 3D image datasets upon passing a readiness assessment conducted using another set of ML based techniques. The automatic annotation of the images may be performed progressively, e.g., by processing a subset or batch of images at a time, and the ML based techniques may be trained to ensure consistency between a forward propagation and a backward propagation.

Abstract: Systems, methods and instrumentalities are described herein for automatically devising and executing a surgical plan associated with a patient in a medical environment, e.g., under the supervision of a medical professional. The surgical plan may be devised based on images of the medical environment captured by one or more sensing devices. A processing device may determine, based on all or a first subset of the images, a patient model that may indicate a location and a shape of an anatomical structure of the patient and determine, based on all or a second subset of the images, an environment model that may indicate a three-dimensional (3D) spatial layout of the medical environment. The surgical plan may be devised based on the patient model and the environment model, and may indicate at least a movement path of a medical device towards the anatomical structure of the patient.

Abstract: A method for surgery planning is provided. The method may include: generating a surgery video displaying a process of a virtual surgery performed by a virtual surgical robot on a virtual patient based on a surgery plan relating to a robotic surgery to be performed on a patient by a surgical robot; transmitting the surgery video to a display component of an XR assembly used to render the surgery video for display to a user; recording one or more actions that the user performed on the virtual surgery based on user input received via an input component of the XR assembly; modifying the surgery plan based on the one or more recorded actions; and causing the surgical robot to implement the robotic surgery on the patient according to the modified surgery plan.

Abstract: A system and method for visualizing anatomical structure of a patient during a surgery are provided. An anatomical image and a first optical image of a patient may be obtained. The anatomical image may be captured by performing a medical scan on the patient before a surgery of the patient, and the first optical image may be captured during the surgery. A target image may be generated by combining the anatomical image and the first optical image. The target image may be rendered based on a viewpoint of a target operator of the surgery. A display device may be directed to display the rendered target image.

Abstract: The 3D pose of a person may be estimated by triangulating 2D representations of body keypoints (e.g., joint locations) of the person. The triangulation may leverage various metrics such as confidence scores associated with the 2D representations of a keypoint and/or temporal consistency between multiple 3D representations of the keypoint. The 2D representations may be arranged into groups, a candidate 3D representation may be determined for each group, taking into account of the confidence score of each 2D representation in the group, and the candidate 3D representation that has the smallest error may be used to represent the keypoint. Other 3D representation(s) of the keypoint determined from images taken at different times may be used to refine the 3D representation of the keypoint.

Abstract: The present disclosure provides a method for surgical automation. The method may include: obtaining video data generated in a surgical process using a camera; identifying a surgical stage in the surgical process based on the video data; and triggering an activation of a surgical equipment used for a surgical operation and/or providing a guidance of the surgical operation based on the surgical stage.

Abstract: The present disclosure provides systems and methods for medical assistant. The systems may obtain position information of a target inside a subject during an operation. The systems may also determine depth information of the target with respect to an operational region of the subject based on the position information. The systems may further direct an optical projection device to project an optical signal representing the depth information on a surface of the subject.

Abstract: A two-dimensional (2D) or three-dimensional (3D) representation of a patient may be provided (e.g., as part of a user interface) to enable interactive viewing of the patient's medical records. A user may select one or more areas of the patient representation. In response to the selection, at least one anatomical structure of the patient that corresponds to the selected areas may be identified based on the user selection. Medical records associated with the at least one anatomical structure of the patient may be determined based on one or more machine-learning models trained for detecting textual or graphical information associated with the at least one anatomical structure in the one or more medical records. The one or more medical records may then be presented, e.g., together with the 2D or 3D representation of the patient.

Abstract: A method and a system for managing healthcare records of a user are provided. The method includes storing an electronic medical record related to the user in form of a non-fungible token (NFT) written to a blockchain, associating a smart contract to the NFT in the blockchain, authorizing a request to access the electronic medical record related to the user based on the defined ownership of the electronic medical record stored in the blockchain, identifying one or more NFTs from the blockchain comprising one or more electronic medical records related to the user based on processing of the identifier information in associated one or more smart contracts therewith, in response to the request, and sending the one or more electronic medical records corresponding to the identified one or more NFTs to a requestor associated with the request.

Abstract: Described herein are systems, methods, and instrumentalities associated with generating a multi-dimensional representation of a medical environment based on images of the medical environments. Various pre-processing and/or post-processing operations may be performed to supplement and/or improve the multi-dimensional representation. These operations may include determining semantic information associated with the medical environment based on the images and adding the semantic information to the multi-dimensional representation in addition to space and time information. The operations may also include anonymizing a person presented in the multi-dimensional representation, adding synthetic views to the multi-dimensional representation, improving the quality of the multi-dimensional representation, etc. The multi-dimensional representation of the medical environment generated using these techniques may allow a user to experience and explore the medical environment, for example, via a virtual reality device.

Abstract: Described herein are systems, methods, and instrumentalities associated with estimating the motions of multiple 3D points in a scene and predicting a view of scene based on the estimated motions. The tasks may be accomplished using one or more machine-learning (ML) models. A first ML model may be used to predict motion-embedding features for a temporal state of a scene, based on motion-embedding features for previous states. A second ML model may be used to predict a motion field representing displacement or deformation of the multiple 3D points from a source time to a target time. Then, a third ML model may be used to predict respective image properties of the 3D points based on their updated locations at the target time and/or a viewing direction. An image of the scene at the target time may then be generated based on the predicted image properties of the 3D points.

Abstract: A system for providing rehabilitation in a virtual environment includes an extended reality (XR) headset to present a first rehabilitation therapy to a patient in a virtual environment. A sensing device is configured to track physical movements of the patient and a processor is configured to receive the sensing data to determine pose information. The processor is configured to determine a performance metric associated with the physical movements and compare the performance metric with a reference metric to determine whether the patient has successfully performed the defined physical movements. The processor is configured to change the first rehabilitation therapy to a second rehabilitation therapy based on a difference between the performance metric and the reference metric upon determining that the patient has unsuccessfully performed the defined physical movements. The system aids the patient by changing the rehabilitation therapies according to the performance of the patient.

Abstract: A non-invasive biometric system includes a processor that is configured to control a scanner, which is configured to scan and capture one or more anatomical images of a body of a target person. The processor is further configured to identify one or more anatomical structures in the captured one or more anatomical images and extract anatomical features for the identified one or more anatomical structures. The processor is further configured to register the extracted anatomical features for the identified one or more identified anatomical structures to a posture and an external appearance of the target person. The processor is further configured to encode and utilize the extracted anatomical features as biometric data, which is unique for the target person, and may be used for authentication of the target person.

Abstract: System and method for differentiable networks trainable to learn an optimized query of a 3D model database used for object recognition includes training a first differentiable network configured as a differentiable renderer by generating 2D images from 3D models of a first object of a dissimilar second object while optimizing rendering parameters for producing 2D images by gradient descent of a first triple loss function. Visual variation among the images is maximized. A second differentiable network configured as a convolutional neural network defined by a regression function is trained by generating searchable feature vectors of the 2D images. The feature vectors are determined using optimized neural network parameters determined by gradient descent of a second triple loss function to achieve high correlation to an input image of the first object and low correlation to images of the second object.

Abstract: System and method for differentiable networks trainable to learn an optimized query of a 3D model database used for object recognition includes training a first differentiable network configured as a differentiable renderer by generating 2D images from 3D models of a first object of a dissimilar second object while optimizing rendering parameters for producing 2D images by gradient descent of a first triple loss function. Visual variation among the images is maximized. A second differentiable network configured as a convolutional neural network defined by a regression function is trained by generating searchable feature vectors of the 2D images. The feature vectors are determined using optimized neural network parameters determined by gradient descent of a second triple loss function to achieve high correlation to an input image of the first object and low correlation to images of the second object.

Abstract: A method of removing noise from a depth image includes presenting real-world depth images in real-time to a first generative adversarial neural network (GAN), the first GAN being trained by synthetic images generated from computer assisted design (CAD) information of at least one object to be recognized in the real-world depth image. The first GAN subtracts the background in the real-world depth image and segments the foreground in the real-world depth image to produce a cleaned real-world depth image. Using the cleaned image, an object of interest in the real-world depth image can be identified via the first GAN trained with synthetic images and the cleaned real-world depth image. In an embodiment the cleaned real-world depth image from the first GAN is provided to a second GAN that provides additional noise cancellation and recovery of features removed by the first GAN.

Abstract: The disclosure relates to a method how to recover an object from a cluttered image. The disclosure also relates to a computer program product and a computer-readable storage medium including instructions which, when the program is executed by a computer, cause the computer to carry out the acts of the mentioned method. Further, the disclosure relates to methods how to train components of a recognition system for recovering an object from such a cluttered image. In addition, the disclosure relates to such a recognition system.

Abstract: A method for training a generative network that is configured for converting cluttered images into a representation of the synthetic domain and a method for recovering an object from a cluttered image.

Abstract: The disclosure relates to a method how to recover an object from a cluttered image. The disclosure also relates to a computer program product and a computer-readable storage medium including instructions which, when the program is executed by a computer, cause the computer to carry out the acts of the mentioned method. Further, the disclosure relates to methods how to train components of a recognition system for recovering an object from such a cluttered image. In addition, the disclosure relates to such a recognition system.

Abstract: A system and method for generating realistic depth images by enhancing simulated images rendered from a 3D model, include a rendering engine configured to render noiseless 2.5D images by rendering various poses with respect to a target 3D CAD model, a noise transfer engine configured to apply realistic noise to the noiseless 2.5D images, and a background transfer engine configured to add pseudo-realistic scenedependent backgrounds to the noiseless 2.5D images. The noise transfer engine is configured to learn noise transfer based on a mapping, by a first generative adversarial network (GAN), of the noiseless 2.5D images to real 2.5D scans generated by a targeted sensor. The background transfer engine is configured to learn background generation based on a processing, by a second GAN, of output data of the first GAN as input data and corresponding real 2.5D scans as target data.