Abstract: A method, system, medium, and implementations for computer-aided preoperative surgical planning are described. Input data acquired with respect to a part of a patient is received by the system. The part corresponds to an organ, e.g., lung, of the patient to be operated on and includes one or more lesions to be removed during an operation. Then, an anatomic 3D model of the part of the patient is generated. Based on the generated anatomic 3D model, a preoperative plan for linear-cutting stapler resection of the one or more lesions from the organ to be carried out during the operation is obtained. The stapler cartridge size and the staple length are estimated based on the preoperative plan. Further, the resection based on the preoperative plan is visualized.

Abstract: The present teaching relates to obtaining three-dimensional (3D) information in a contactless manner using a one-dimensional (1D) distance sensor. The 1D distance sensor is associated with an origin from where a laser beam emitted in an orientation, where the origin/orientation are calibrated with respect to a marker attached thereon and tracked. In a surgery, the calibrated 1D distance sensor is tracked by a camera via the marker attached thereon. When an emitted laser beam hits at a location on a patient, the 3D coordinate thereof in the tracker coordinate system is determined based on tracked marker, the origin/orientation of the 1D distance sensor, and a distance between the origin of the 1D distance sensor and the location.

Abstract: The present teaching relates to method, system, medium, and implementations for estimating 3D coordinate of a 3D virtual model. Two pairs of feature points are obtained. Each of the pairs includes a respective 2D feature point on an organ observed in a 2D image, acquired during a medical procedure, and a respective corresponding 3D feature point from a 3D virtual model, constructed for the organ prior to the procedure based on a plurality of images of the organ. The first and the second 3D feature points have different depths. A 3D coordinate of a 3D feature point is determined based on the pairs of feature points so that a projection of the 3D virtual model from the 3D coordinate substantially matches the organ observed in the 2D image.

Abstract: The present teaching relates to automated robot base location determination. An instrument access map is obtained with respect to a surgical instrument for performing a surgical operation on an organ and defines a surface area on the organ with cut points that form a surgery trajectory. Information about a robot and a surgical environment is received and used to generate a robot access map with multiple robot base locations. The robot is for controlling the surgical instrument to perform the surgical operation along the surgery trajectory. A robot base location is selected based on evaluation parameters derived for each of the robot base locations. Control signals are generated for configuring the robot at the selected robot base location to facilitate the surgical operation.

Abstract: A method, system, medium, and implementations for computer-aided preoperative surgical planning are described. Input data acquired with respect to a part of a patient is received by the system. The part corresponds to an organ, e.g., lung, of the patient to be operated on and includes one or more lesions to be removed during an operation. Then, an anatomic 3D model of the part of the patient is generated. Based on the generated anatomic 3D model, a preoperative plan for linear-cutting stapler resection of the one or more lesions from the organ to be carried out during the operation is obtained. The stapler cartridge size and the staple length are estimated based on the preoperative plan. Further, the resection based on the preoperative plan is visualized.

Abstract: The present teaching relates to determination of an optimal set of supplying vessel branches for delivering radioactive material to a target region. A centerline representation for blood vessels is obtained based on a 3D image capturing an organ with a target region and a non-target region. Supplying centerline segments, each representing a supplying vessel branch, are identified based on connected supplying centerline points that cover some parts of the target region. An optimal set of supplying vessel branches are selected that satisfy a specified coverage to the target region with a least coverage to the non-target region.

Abstract: A method, implemented on a machine having at least one processor, storage, and a communication platform capable of connecting to a network for characterizing a tumor embedded inside a medium, comprising: measuring at least one of a mechanical property and a spectral property of the tumor.

Abstract: The present teaching relates to surgical position marking. A 3D model for an organ includes cut points forming a surgical trajectory. Each cut point has a 3D coordinate and a surface norm in the model space. When projected into a workspace, a mapped cut point is created with a mapped 3D coordinate and a mapped surface norm in the workspace. With a surgical instrument with a tip and a force sensor attached thereto, some mapped cut points are marked along a direction determined based on a reaction force sensed by the force sensor when the tip touches the cut point.

Abstract: The present teaching relates to automated trocar/robot base location determination. An input relates to a surgical operation with a 3D model for an organ including cut points thereon forming a surgical trajectory. A surgical instrument is controlled by a robot to reach the cut points to carry out the surgical operation. Candidate combinations of insertion location for inserting the surgical instrument and base location for deploying the robot are generated. One of the candidate combinations is identified based on evaluation vectors associated therewith. Each evaluation vector provides assessment information on characteristics and spatial relationships of the respective candidate locations. The selected combination of locations is used for the surgical operation.

Abstract: The present teaching relates to method, system, medium, and implementations for region boundary identification and overlay display. An input associated with a 3D object is received. A 3D volumetric model is obtained for the 3D object with multiple regions, each of which includes multiple labeled voxels. A mesh representation is generated for the surface of the 3D object. Region boundaries on surface of the 3D object are identified based on the 3D volumetric model and the mesh representation. The 3D object is rendered, and the region boundaries are overlaid on the rendered 3D object.

Abstract: The present teaching relates to generating a point cloud of a target organ using a camera and a laser emitter inserted into a patient's body near the target organ and tracked by a tracker. The laser emitter emits, at tracked moving locations, laser beams that hit the target organ at multiple points which are captured by the camera in an image. A three-dimensional (3D) coordinate for each point is determined based on a laser line equation, connecting the laser emitter and the point, and an image line equation, connecting the camera and the point. The 3D coordinates for the multiple points are used to generate the point cloud.

Abstract: The present teaching relates to estimating 3D pose of an ultrasound probe. An ultrasound image is acquired by an ultrasound probe deployed at a three-dimensional (3D) probe pose during a medical procedure directed to a target organ. A mask for each two-dimensional (2D) anatomical structure is identified and a corresponding label is estimated from the ultrasound image to generate an ASM/label pair. If a sequence of prior 3D probe poses does not exist, an ASM/label representation for the ultrasound image is generated based on the ASM/label pairs from the ultrasound image and used to estimate the 3D pose of the probe via an ASM-pose mapping model. If the sequence of prior 3D probe poses exists, the 3D probe pose is predicted based on the sequence of prior 3D probe poses and an ASM/label representation is accordingly generated based on a virtual ultrasound image created based on the predicted 3D probe pose.

Abstract: The present teaching relates to a method and system for path planning. A target is tracked via one or more sensors. Information of a desired pose of an end-effector with respect to the target and a current pose of the end-effector is obtained. Also, a minimum distance permitted between an arm including the end-effector and each of at least one obstacle identified between the current pose of the end-effector and the target is obtained. A weighting factor previously learned is retrieved and a cost based on a cost function is computed in accordance with a weighted smallest distance between the arm including the end-effector and the at least one obstacle, wherein the smallest distance is weighted by the weighting factor. A trajectory is computed from the current pose to the desired pose by minimizing the cost function.

Abstract: The present teaching is directed to estimating 3D camera pose based on 2D features detected from a 2D image. Virtual 3D camera poses are generated with respect to a 3D model for a target organ and associated anatomical structures. Virtual 2D images are created by projecting the 3D model from perspectives determined based on the virtual 3D camera poses. Each virtual 2D image includes 2D projected target organ and/or 2D structures of some 3D anatomical structures visible from a corresponding perspective. 2D feature/camera pose mapping models are then accordingly obtained based on 2D features extracted from the virtual 2D images and the corresponding virtual 3D camera poses, where the 2D features include a 2D ridge line projected from a 3D ridge on the target organ represented in the 3D model.

Abstract: The present teaching relates to determining an optimal robot base location that maximizes robot manipulability. A trocar location is determined for inserting a surgical instrument manipulated by a robot to reach a target organ. With respect to the trocar location, candidate base locations are generated, each of which is a location to deploy the robot for manipulating the surgical instrument through the trocar location. Each candidate base location is evaluated based on criteria indicative of the robot's manipulability of the surgical instrument with respect to the target organ. An optimal base location is selected from the candidate base locations based on the evaluation result.

Abstract: System, method, medium, and implementation of registering an ultrasound probe in a laparoscopic ultrasound procedure and application thereof is disclosed. The two-dimensional (2D) location of the ultrasound probe in a 2D laparoscopic (LP) image is detected, where the LP camera is previously calibrated in a three-dimensional (3D) space. The 3D pose of the ultrasound probe as deployed is estimated and registered in the 3D space based on the detected 2D location of the ultrasound probe and an ultrasound model for the ultrasound probe.

Abstract: Method, system, medium, and implementation for robot-guided instrument insertion. A preplanned path between a three-dimensional (3D) entry pose on skin of a patient to a 3D pose of a target is generated for a surgery with a target inside the patient. The preplanned path is used by a robot as guidance to insert a surgical instrument from the 3D entry pose to reach the 3D pose of the target. During the surgery, a next pose is determined based on a current pose of the surgical instrument and the preplanned path as well as spatial relationship to surrounding anatomical structures and is used to move the surgical instrument thereto. An updated current pose of the instrument is then obtained via tracking when the robot is advancing the surgical instrument to the next pose. The process repeats until the instrument reaches the target pose.

Abstract: The present teaching relates to automated trocar/robot base location determination. An input related to a surgical operation includes a three-dimensional (3D) model for an organ with cut points on the organ forming a surgical trajectory. A surgical instrument is controlled by a robot to reach the cut points to carry out the operation. An insertion location for inserting the surgical instrument is identified. Based on the identified insertion location, a robot base location is determined with respect to the identified insertion location to deploy the robot for controlling the surgical instrument to reach the cut points. Signals for configuring the surgical setting are then generated based on the insertion location for the surgical instrument and the base location for the robot.

Abstract: Method and system for estimating 3D camera pose based on 2D features. 3D virtual camera poses are generated, each of which is used to determine a perspective to project a 3D model of a target organ to create a 2D image of the target organ. 2D features are extracted from each 2D image and paired with the corresponding 3D virtual camera pose to represent a mapping. A 2D feature-camera pose mapping model is obtained based on the pairs. Input 2D features extracted from a real-time 2D image of the target organ are used to map, via the 2D feature-camera pose mapping model, to a 3D pose estimate of a laparoscopic camera, which is then refined to derive an estimated 3D camera pose of the laparoscopic camera via differential rendering of the 3D model with respect to the 3D pose estimate.

Abstract: The present teaching relates to automated generation of a surgical tool-based visual guide. Two-dimensional (2D) images capturing anatomical structures and a surgical instrument therein are provided during a surgery. The type and pose of a tool attached to the surgical instrument are detected based on the 2D images. Focused information is determined based on the type and pose of the detected tool and is used to generate a visual guide to assist a user to perform a surgical task using the tool.