Abstract: Robotic and/or measurement devices, systems, and methods for telesurgical and other applications employ input devices operatively coupled to tools so as to allow a system user to manipulate tissues and other structures being measured. The system may make use of three dimensional position information from stereoscopic images. Two or more discrete points can be designated in three dimensions so as to provide a cumulative length along a straight or curving structure, an area measurement, a volume measurement, or the like. The discrete points may be identified by a single surgical tool or by distances separating two or more surgical tools, with the user optionally measuring a structure longer than a field of view of the stereoscopic image capture device by walking a pair of tools “hand-over-hand” along the structure.

Abstract: A computer implemented method for fusing position and range measurements to determine the position of at least one node of a plurality of distributed nodes is disclosed.

Abstract: The present disclosure relates to calibration assemblies and methods for use with an imaging system, such as an endoscopic imaging system. A calibration assembly includes: an interface for constraining engagement with an endoscopic imaging system; a target coupled with the interface so as to be within the field of view of the imaging system, the target including multiple of markers having calibration features that include identification features; and a processor configured to identify from first and second images obtained at first and second relative spatial arrangements between the imaging system and the target, respectively, at least some of the markers from the identification features, and using the identified markers and calibration feature positions within the images to generate calibration data.

Abstract: The present disclosure relates to systems, methods, and tools for tool tracking using image-derived data from one or more tool located reference features. A method includes: directing illuminating light from a light source onto a robotic surgical tool within a patient body, wherein the tool includes a plurality of primitive features having known positions on the tool, and wherein each feature includes a spherical reflective surface; capturing stereo images of the plurality of primitive features when the tool is within the patient body, wherein the stereo images are captured by an image capture device adjacent the illumination source so that the illumination light reflected from the imaged primitive features toward the image capture device substantially aligns with spherical centers of the surfaces of the imaged primitive features, and determining a position for the tool by processing the stereo images so as to locate the spherical centers of the imaged primitive features by using the reflected light.

Abstract: The present disclosure relates to systems, methods, and tools for tool tracking using image-derived data from one or more tool-located reference features. A method includes: capturing a first image of a tool that includes multiple features that define a first marker, where at least one of the features of the first marker includes an identification feature; determining a position for the first marker by processing the first image; determining an identification for the first marker by using the at least one identification feature by processing the first image; and determining a tool state for the tool by using the position and the identification of the first marker.

Abstract: Systems, methods, and devices are used to match images. Points of interest from a first image are identified for matching to a second image. In response to the identified points of interest, regions and features can be identified and used to match the points of interest to a corresponding second image or second series of images. Regions can be used to match the points of interest when regions of the first image are matched to the second image with high confidence scores, for example above a threshold. Features of the first image can be matched to the second image, and these matched features may be used to match the points of interest to the second image, for example when the confidence scores for the regions are below the threshold value. Constraint can be used to evaluate the matched points of interest, for example by excluding bad points.

Abstract: Methods of and a system for providing a visual representation of force information in a robotic surgical system. A real position of a surgical end effector is determined. A projected position of the surgical end effector if no force were applied against the end effector is also determined. Images representing the real and projected positions are output superimposed on a display. The offset between the two images provides a visual indication of a force applied to the end effector or to the kinematic chain that supports the end effector. In addition, tissue deformation information is determined and displayed.

Abstract: An apparatus is configured to show telestration in 3-D to a surgeon in real time. A proctor is shown one side of a stereo image pair, such that the proctor can draw a telestration line on the one side with an input device. Points of interest are identified for matching to the other side of the stereo image pair. In response to the identified points of interest, regions and features are identified and used to match the points of interest to the other side. Regions can be used to match the points of interest. Features of the first image can be matched to the second image and used to match the points of interest to the second image, for example when the confidence scores for the regions are below a threshold value. Constraints can be used to evaluate the matched points of interest, for example by excluding bad points.

Abstract: Methods of and a system for providing force information for a robotic surgical system. The method includes storing first kinematic position information and first actual position information for a first position of an end effector; moving the end effector via the robotic surgical system from the first position to a second position; storing second kinematic position information and second actual position information for the second position; and providing force information regarding force applied to the end effector at the second position utilizing the first actual position information, the second actual position information, the first kinematic position information, and the second kinematic position information. Visual force feedback is also provided via superimposing an estimated position of an end effector without force over an image of the actual position of the end effector. Similarly, tissue elasticity visual displays may be shown.

Abstract: The present invention relates to a system and methodology facilitating automated manufacturing processes in an industrial controller environment. An automation system is provided for automated industrial processing. The system includes an equipment phase object that is executed by a controller engine, wherein the equipment phase object can be accessible from internal instructions within the controller and/or from external instructions directed to the controller such as from a server or another controller across a network connection. A sequencing engine operates with the equipment phase object to facilitate automated industrial processing. The sequencing engine can be adapted to various industrial standards or in accordance with other state type models.

Abstract: The present invention provides a system and method for detecting and tracking a moving object. First, robust change detection is applied to find initial candidate regions in consecutive frames. These initial detections in consecutive frames are stacked to produce space-time bands which are extracted by Hough transform and entropy minimization based band detection algorithm.

Abstract: A method and/or system for tracking objects, such as humans, over a wide area (that is, over an area that is delineated by a large spatial domain and/or a long-duration temporal domain) is provided. Such tracking is facilitated by processing, in real-time, near real-time or otherwise contemporaneous with receiving, images captured by each of a plurality or network of slightly overlapping stereo sensors, such as stereo cameras. The method includes and the apparatus is adapted for obtaining a plurality of local-track segments, wherein the plurality of local-track segments correspond to an object captured in images taken by a respective plurality of stereo sensors; and combining the local-track segments to form a global track.

Abstract: An illumination channel, a stereoscopic optical channel and another optical channel are held and positioned by a robotic surgical system. A first capture unit captures a stereoscopic visible image from the first light from the stereoscopic optical channel while a second capture unit captures a fluorescence image from the second light from the other optical channel. An intelligent image processing system receives the captured stereoscopic visible image and the captured fluorescence image and generates a stereoscopic pair of fluorescence images. An augmented stereoscopic display system outputs a real-time stereoscopic image comprising a three-dimensional presentation of a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images.

Abstract: A robotic surgical system positions and holds an endoscope. A visible imaging system is coupled to the endoscope. The visible imaging system captures a visible image of tissue. An alternate imaging system is also coupled to the endoscope. The alternate imaging system captures a fluorescence image of at least a portion of the tissue. A stereoscopic video display system is coupled to the visible imaging system and to the alternate imaging system. The stereoscopic video display system outputs a real-time stereoscopic image comprising a three-dimensional presentation of a blend of a fluorescence image associated with the captured fluorescence image, and the visible image.

Abstract: An endoscope with a stereoscopic optical channel is held and positioned by a robotic surgical system. A capture unit captures (1) a visible first image and (2) a visible second image combined with a fluorescence second image from the light. An intelligent image processing system receives (1) the visible first image and (2) the visible second image combined with the fluorescence second image and generates at least one fluorescence image of a stereoscopic pair of fluorescence images and a visible second image. An augmented stereoscopic display system outputs a real-time stereoscopic image including a three-dimensional presentation including in one eye, a blend of the at least one fluorescence image of a stereoscopic pair of fluorescence images and one of the visible first and second images; and in the other eye, the other of the visible first and second images.

Abstract: A computer implemented method for fusing position and range measurements to determine the position of at least one node of a plurality of distributed nodes is disclosed.

Abstract: A synthetic representation of a robot tool for display on a user interface of a robotic system. The synthetic representation may be used to show the position of a view volume of an image capture device with respect to the robot. The synthetic representation may also be used to find a tool that is outside of the field of view, to display range of motion limits for a tool, to remotely communicate information about the robot, and to detect collisions.

Abstract: A method and/or system for tracking objects, such as humans, over a wide area (that is, over an area that is delineated by a large spatial domain and/or a long-duration temporal domain) is provided. Such tracking is facilitated by processing, in real-time, near real-time or otherwise contemporaneous with receiving, images captured by each of a plurality or network of slightly overlapping stereo sensors, such as stereo cameras. The method includes and the apparatus is adapted for obtaining a plurality of local-track segments, wherein the plurality of local-track segments correspond to an object captured in images taken by a respective plurality of stereo sensors; and combining the local-track segments to form a global track.

Abstract: A sentient system combines detection, tracking, and immersive visualization of a cluttered and crowded environment, such as an office building, terminal, or other enclosed site using a network of stereo cameras. A guard monitors the site using a live 3D model, which is updated from different directions using the multiple video streams. As a person moves within the view of a camera, the system detects its motion and tracks the person's path, it hands off the track to the next camera when the person goes out of that camera's view. Multiple people can be tracked simultaneously both within and across cameras, with each track shown on a map display. The track system includes a track map browser that displays the tracks of all moving objects as well as a history of recent tracks and a video flashlight viewer that displays live immersive video of any person that is being tracked.

Abstract: A unified approach, a fusion technique, a space-time constraint, a methodology, and system architecture are provided. The unified approach is to fuse the outputs of monocular and stereo video trackers, RFID and localization systems and biometric identification systems. The fusion technique is provided that is based on the transformation of the sensory information from heterogeneous sources into a common coordinate system with rigorous uncertainties analysis to account for various sensor noises and ambiguities. The space-time constraint is used to fuse different sensor using the location and velocity information. Advantages include the ability to continuously track multiple humans with their identities in a large area. The methodology is general so that other sensors can be incorporated into the system. The system architecture is provided for the underlying real-time processing of the sensors.