Abstract: The present teaching relates to method, system, medium, and implementations for fusing a 3D virtual model with a 2D image associated with an organ of a patient. A key-pose is determined as an approximate position and orientation of a medical instrument with respect to the patient's organ. Based on the key-pose, an overlay is generated on a 2D image of the patient's organ, acquired by the medical instrument, by projecting the 3D virtual model on to the 2D image. A pair of feature points includes a 2D feature point from the 2D image and a corresponding 3D feature point from the 3D virtual model. The 3D coordinate of the 3D feature point is determined based on the 2D coordinate of the 2D feature point. The depth of the 3D coordinate is on a line of sight of the 2D feature point and is determined so that the projection of the 3D virtual model from the depth creates an overlay approximately matching the organ observed in the 2D image.

Abstract: The present teaching relates to a method and system for path planning. Information of a current pose of a robotic arm having a plurality of operable segments is obtained. The information includes a plurality of values, each of which corresponds to an angle formed between consecutive operable segments of the robotic arm. A desired pose where the robotic arm needs to reach is also obtained. An angle step-value is computed for the current pose of the robotic arm based on a function of a distance between the current pose and the desired pose, wherein the angle step value is to be used to determine a plurality of candidate next poses of the plurality of operable segments. One or more of candidate next poses is selected based on at least one criterion, and a trajectory is determined from the current pose to the desired pose based on the selected next poses.

Abstract: The present teaching relates to method, system, medium, and implementations for estimating 3D coordinate of a 3D virtual model. Two pairs of feature points are obtained. Each of the pairs includes a respective 2D feature point on an organ observed in a 2D image, acquired during a medical procedure, and a respective corresponding 3D feature point from a 3D virtual model, constructed for the organ prior to the procedure based on a plurality of images of the organ. The depths of the first and the second 3D feature points are substantially the same. A first 3D coordinate of the first 3D feature point and a second 3D coordinate of the second 3D feature point are automatically determined based on the pairs of feature points so that a first distance between the determined first 3D coordinate and the determined second 3D coordinate equals to a second distance between a first actual 3D coordinate of the first 3D feature point and a second actual 3D coordinate of the second 3D feature point in the 3D virtual model.

Abstract: The present teaching relates to method, system, medium, and implementations for robot path planning. Depth data of obstacles, acquired by depth sensors deployed in a 3D robot workspace and represented with respect to a sensor coordinate system, is transformed into depth data with respect to a robot coordinate system. The 3D robot workspace is discretized to generate 3D grid points representing a discretized 3D robot workspace. Based on the depth data with respect to the robot coordinate system, binarized values are assigned to at least some of 3D grid points to generate a binarized representation for the obstacles present in the 3D robot workspace. With respect to one or more sensing points associated with a part of a robot, it is determined whether the part is to collide with any obstacle. Based on the determining, a path is planned for the robot to move along while avoiding any obstacle.

Abstract: The present teaching relates to a method and system for path planning. A target is tracked via one or more sensors. Information of a desired pose of an end-effector with respect to the target and a current pose of the end-effector is obtained. Also, a minimum distance permitted between an arm including the end-effector and each of at least one obstacle identified between the current pose of the end-effector and the target is obtained. A weighting factor previously learned is retrieved and a cost based on a cost function is computed in accordance with a weighted smallest distance between the arm including the end-effector and the at least one obstacle, wherein the smallest distance is weighted by the weighting factor. A trajectory is computed from the current pose to the desired pose by minimizing the cost function.

Abstract: The present disclosure relates to robot path planning. Depth information of a plurality of obstacles in an environment of a robot are obtained at a first time instance. A static distance map is generated based on the depth information. A path is computed for the robot based on the static distance map. At a second time instant, depth information of one or more obstacles is obtained. A dynamic distance map is generated based on the one or more obstacles, wherein for each obstacle that satisfies a condition: a vibration range of the obstacle is computed based on a position of the obstacle and the static distance map, and the obstacle is classified as a dynamic obstacle or a static obstacle based on a criterion associated with the vibration range. A repulsive speed of the robot is computed based on the dynamic distance map to avoid the dynamic obstacles.

Abstract: The present teaching relates to surgical procedure planning. In one example, at least one 3D object contained in a 3D volume is rendered on a display screen. The at least one 3D object includes a 3D object corresponding to an organ. First information related to a 3D pose of a surgical instrument positioned with respect to the at least one 3D object is received from a user. A 3D representation of the surgical instrument is rendered in the 3D volume based on the first information. Second information related to a setting of the surgical instrument is received from the user. A 3D treatment zone in the 3D volume with respect to the at least one 3D object is estimated based on the first and second information. The 3D treatment zone in the 3D volume is visualized on the display screen. Controls associated with the 3D representation of the surgical instrument and/or the 3D treatment zone are provided to facilitate the user to dynamically adjust the 3D treatment zone via the controls.

Abstract: Systems and methods relating to placement of an ultrasound transducer for a procedure. In a non-limiting embodiment, a 3D environment including images of a body region of a patient, as well as images including a first virtual representation of an ablation needle at a first location and a second virtual representation of an ultrasound transducer at a second location is rendered on a display device. A determination may be made as to whether the first and second virtual representations collide at a first collision point. If so, at least one parameter associated with an orientation and/or position of the second virtual representation may be adjusted. A determination may then be made as to whether or not the first and second virtual representations still collide and, in response to determining that there is no collision, position data indicating the location of the second virtual representation after the adjustment, is stored.

Abstract: The present teaching relates to surgical procedure assistance. In one example, a first image of a patient captured prior to a surgical procedure is received. A treatment plan is generated based on the first image. The treatment plan includes information related to one or more surgical instruments. A second image of the patient captured after the surgical procedure has been initiated is received. The treatment plan is dynamically adjusted based on a pose of any of the one or more surgical instruments identified from the second image. A third image of the patient captured after a lesion is treated by at least one of the surgical instruments based on the adjusted treatment plan is received. Whether a further treatment to the lesion is needed is determined based on the third image. Upon determining a further treatment is needed, an updated treatment plan is dynamically generated based on the third image.

Abstract: Methods, systems, and programs for real-time surgical procedure assistance are provided. A first set of 3D poses of the 3D points on the organ may be received. An electronic organ map built for the organ via pre-surgical medical information may be retrieved. A tissue parameter of the organ may be obtained based on the first set of 3D poses and their corresponding 3D poses from the electronic organ map. A deformation transformation of the electronic organ map may be calculated based on the obtained tissue parameter and the first set of 3D poses during the surgical procedure. The deformed electronic organ map may be projected onto the organ with respect to the first set of 3D poses during the surgical procedure.

Abstract: The present teaching relates to interactive medical image processing for surgical procedure planning. In one example, a three dimensional (3D) image of a kidney is obtained. The three dimensional image is rendered on a display screen. An input is received from a user specifying a location with respect to a representation of the kidney in the rendered three dimensional image. A representation of an instrument is rendered on the display screen based on the location. The instrument is automatically aligned with an infundibulum pathway of calyx at the location with respect to the kidney. A graphical line extension is rendered on the display screen to visualize the alignment of the instrument. One or more measurements related to the kidney are determined based on the location and an anatomical structure of the kidney.

Abstract: The present teaching relates to method and system for aligning a virtual anatomic model. The method generates a virtual model of an organ of a patient, wherein the virtual model includes at least three virtual markers. A number of virtual spheres equal to a the number of virtual markers are generated, wherein the virtual spheres are disposed on the virtual model of the organ of the patient and associated with the virtual markers. A first position of the virtual spheres and the virtual markers is recorded. The virtual spheres are placed to coincide with physical markers disposed on the patient and a second position of the virtual spheres is recorded. A transformation of the virtual spheres and the virtual markers based on the first and second positions is computed and the virtual model of the organ is aligned with the patient based on the computed transformation.

Abstract: The present teaching relates to surgical procedure assistance. In one example, a first volume of air inside a lung is obtained based on a first image of the lung captured prior to a surgical procedure. The lung has a first shape on the first image. A second volume of air deflated from the lung is determined based on a second image of the lung captured during the surgical procedure. A second shape of the lung is estimated based on the first shape of the lung and the first air volume inside the lung and second volume of air deflated from the lung. A surgical plan is updated based on the estimated second shape of the lung.

Abstract: The success of percutaneous radiofrequency ablation mainly depends on the accuracy of the needle insertion, making it possible to destroy the whole tumor, while avoiding damages on other organs and minimizing risks of a local recurrence. This invention presents a simulated 3D environment for user to interactively place a treatment probe to a target position.

Abstract: The present teaching relates to surgical procedure assistance. In one example, a first image of an organ having a lesion is obtained prior to a surgical procedure. Information related to a pose of a surgical instrument at a first location with respect to the lesion is received from a sensor coupled with the surgical instrument. A visual environment having the lesion and the surgical instrument rendered therein is generated based on the first image and the information received from the sensor. A second image of the organ is obtained when the surgical instrument is moved to a second location with respect to the lesion during the surgical procedure. The second image captures the lesion and the surgical instrument. The pose of the surgical instrument and the lesion rendered in the visual environment are adjusted based, at least in part, on the second image.

Abstract: The present teaching relates to surgical procedure assistance. In one example, a first set of positions of a plurality of sensors are obtained in an image captured prior to a surgical procedure. The plurality of sensors are coupled with a patient's body. One or more second sets of positions of the plurality of sensors are obtained based on information from a tracking device associated with the plurality of sensors. The one or more second sets of positions change in accordance with movement of the patient's body. One or more similarity measures are calculated between the first set of positions and each of the one or more second sets of positions of the plurality of sensors. A timing at which the surgical procedure initiates is determined based on the one or more similarity measures.

Abstract: The present disclosure relates to a system and method of providing surgical assistance. The method includes obtaining a 3D space including an organ and a lesion located within the organ. Based on a shape of the organ, a grid is overlaid on a surface of the organ. The lesion is projected at a first location on an overlaid grid. A graphical tool disposed in a first orientation is positioned at the first location in the 3D space, and a plurality of parameter values of the lesion corresponding to the first orientation of the graphical tool are displayed in the 3D space. The graphical tool is manipulated to be disposed in a second orientation at a second location on the overlaid grid, and the plurality of parameter values of the lesion corresponding to the second orientation of the graphical tool are displayed in the 3D space.

Abstract: The present teaching relates to surgical procedure assistance. In one example, a plurality of similarity measures are determined between a first set of positions and a plurality of second sets of positions, respectively. The first set of positions is obtained with respect to a plurality of sensors coupled with a patient in an image captured prior to a surgical procedure. The plurality of second sets of positions are obtained from the plurality of sensors and change in accordance with movement of the patient. A target lesion is segmented in the image captured prior to the surgical procedure to obtain a lesion display object. The lesion display object is duplicated to generate a plurality of lesion display objects. The plurality of lesion display objects are presented on a display screen so that a distance between the plurality of lesion display objects changes in accordance with the plurality of the similarity measures.

Abstract: A method for labeling vessel branches forming first and second vessel systems. A 3D image is segmented to obtain vessel branches. First and second root points are then determined. Starting from the first root point, the vessel branches are traced to derive a tracing path and a break point is determined, that separates the tracing path into two portions. A region of interest in the 3D image is determined with respect to the break point, in which center lines are assigned to the first or second vessel system. A 3D cutting structure is generated based on distances measured from points on the center lines to the 3D cutting structure. Graph representations are constructed for the first and second vessel systems. The vessel branches are labeled with different labels based on the graph representations.

Abstract: The present teaching relates to surgical procedure assistance. In one example, a first volume of air inside a lung is obtained based on a first image of the lung captured prior to a surgical procedure. The lung has a first shape on the first image. A second volume of air deflated from the lung is determined based on a second image of the lung captured during the surgical procedure. A second shape of the lung is estimated based on the first shape of the lung and the first air volume inside the lung and second volume of air deflated from the lung. A surgical plan is updated based on the estimated second shape of the lung.