Abstract: A medical robotic system includes a viewer for displaying an image of a work site, a gaze tracker for tracking a gaze point of a user on the viewer, and a processor. The processor is configured to: draw an area or volume defining shape, overlaid on the image of the work site, in a position determined by the gaze tracker; assign a fixed virtual constraint to the shape and constrain movement of a robotic tool according to the fixed virtual constraint; receive a user selected action command selecting an image of patient anatomy; and superimpose the selected image of the patient anatomy over the image of the work site within the shape. The selected image of the patient anatomy is registered to the image of the work site.

Abstract: A robotic system may comprise a first robotic arm operatively coupleable to a first tool. The first tool has a first working end. The system may also comprise an image capture device, a display, and a processor. The processor may be configured to cause an image of a work site, which was captured by the image capture device from a perspective of an image reference frame, to be displayed on the display. The image of the work site includes an image of the first working end of the first tool. The processor may also determine a position of the first working end of the first tool in the image of the work site and render a tool information overlay at the position of the first working end of the first tool in the image of the work site. The tool information overlay visually indicates state information for the first tool. The processor may also change the tool information overlay while the first tool is in a first operational state by changing a brightness of the tool information overlay.

Abstract: The systems and methods described can include approaches to calibrate head-mounted displays for improved viewing experiences. Some methods include receiving data of a first target image associated with an undeformed state of a first eyepiece of a head-mounted display device; receiving data of a first captured image associated with deformed state of the first eyepiece of the head-mounted display device; determining a first transformation that maps the first captured image to the image; and applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

Abstract: An exemplary method includes receiving a frame of raw pixels captured by an image capture sensor, a particular location in the frame of raw pixels including a captured color pixel value of a first color component of a first color space; generating, for the particular location in the frame of raw pixels, a vertical color difference signal and a horizontal color difference signal using information from the frame of raw pixels; determining that the particular location is missing a color pixel value of a second color component of the first color space; generating a reconstructed color pixel value of the second color component based on the vertical color difference signal and the horizontal color difference signal; and transforming the vertical color difference signal and the horizontal color difference signal into chrominance pixel values in a second color space that is different from the first color space.

Abstract: A computing platform (10), a method, and an apparatus (20) for spiking neural network (SNN) learning and simulation are provided. The computing platform (10) includes a neuron dynamics simulation module (11), a neuron conversion module (12), an SNN construction and weight learning module (13), and a neural network level parameter and weight access module (14). The neuron dynamics simulation module (11) simulates changing features of neurons. The neuron conversion module (12) performs operations on a calculation graph. The SNN construction and weight learning module (13) updates and iterates connection weights. The neural network level parameter and weight access module (14) stores overall network detail parameters.

Abstract: An example method includes causing an illuminator to simultaneously illuminate tissue with a first visible color illumination component, a second visible color illumination component, and a fluorescence excitation illumination component; and causing a display unit to display, based on the illumination, a display scene that includes: a reduced color scene component corresponding to the first and second visible color illumination components, and a highlighted scene component corresponding to the fluorescence excitation illumination component.

Abstract: A calibration assembly may be used with an endoscopic imaging system having a field of view. The calibration assembly may comprise an interface configured for constraining engagement with the endoscopic imaging system and a target coupled with the interface so as to be within the field of view. The target may include a plurality of calibration features. The calibration assembly may be reconfigurable from a first spatial arrangement between the target and the endoscopic imaging system to a second spatial arrangement between the target and the endoscopic imaging system. The calibration assembly may also comprise a processor configured to obtain, from the endoscopic imaging system, first and second images of at least some of the calibration features of the target at the first and second spatial arrangements. The processor may also generate calibration data for the endoscopic imaging system using the first and second images.

Abstract: The systems and methods described can include approaches to calibrate head-mounted displays for improved viewing experiences. Some methods include receiving data of a first target image associated with an undeformed state of a first eyepiece of a head-mounted display device; receiving data of a first captured image associated with deformed state of the first eyepiece of the head-mounted display device; determining a first transformation that maps the first captured image to the image; and applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

Abstract: A target tracking method and a target tracking system of a spiking neural network based on an event camera are provided. The method includes: acquiring a data stream of asynchronous events in a high dynamic scene of a target by an event camera as input data; dividing the data stream of the asynchronous events into synchronous event frames with millisecond time resolution; training a twin network based on a spiking neural network by a gradient substitution algorithm with a target image as a template image and a complete image as a searched image; and tracking the target by a trained twin network with interpolating a result of feature mapping to up-sample and obtaining the position of the target in an original image. The twin network includes a feature extractor and a cross-correlation calculator.

Abstract: An exemplary method includes a demosaic module receiving a frame of raw pixels having values in a first color space, generating vertical and horizontal color difference signals using information from the frame of raw pixels, generating reconstructed color pixel values for raw pixels that are missing captured color pixel values of a first color component of the first color space, transforming the captured and reconstructed color pixel values of the first color component to brightness pixel values in a second color space, and transforming the vertical and horizontal color difference signals into chrominance pixel values in the second color space.

Abstract: A system may comprise an image capture device to capture an image of a work site. The system may also comprise a processor to determine whether a tool disposed at the work site is energized and determine a first area of the captured image of the work site, including or adjacent to an image of a portion of the tool in the captured image. The processor may also determine a second area of the captured image including a remainder of the captured image that does not include the first area. Conditioned upon determining that the tool is energized, at least one of the first area and the second area of the captured image of the work site may be processed to indicate that the tool in the first area is being energized. The image of the work site with the first area and the second area may be displayed.

Abstract: An example method includes receiving a single frame comprising a first set of pixel data that includes a fluorescence scene component and a second set of pixel data that includes a combination of a visible color component scene and the fluorescence scene component; and generating, based on the first set of pixel data that includes the fluorescence scene component and the second set of pixel data that includes the combination of the visible color component scene and the fluorescence scene component, a display scene.

Abstract: A method for intelligent manual adjustment of a manually rotatable focus control of a camera may comprise determining a depth value of a target area relative to the camera and determining a desirable focus adjustment according to a direction and an amount that a focal point of the camera is to be moved in order to coincide with the depth value of the target area. The method may also comprise assisting an operator of the manually rotatable focus control to the desirable focus adjustment by providing a haptic force on the manually rotatable focus control that decreases as the focal point of the camera moves toward the depth value of the target area and increases as the focal point of the camera moves away from the depth value of the target area.

Abstract: A medical robotic system includes a viewer, a gaze tracker, and a processor programmed to: draw an area or volume defining shape overlaid on an image based on the tracked gaze point after the user has gazed on the tracked gaze point for a programmed period of time; in response to receiving a user-selected action command, assign a fixed virtual constraint to the area or volume defining shape and constrain movement of a robotic tool; map points of the robotic tool in a tool reference frame to a viewer reference frame; determine a closest object to the tracked gaze point is the robotic tool based at least in part on the mapped points; display an object including text identifying the robotic tool, overlaid on the image, proximate to the robotic tool based on determining the robotic tool is the closest object; and perform an action indicated by the object.

Abstract: A robotic system provides user selectable actions associated with gaze tracking according to user interface types. User initiated correction and/or recalibration of the gaze tracking may be performed during the processing of individual of the user selectable actions.

Abstract: A system comprises a first robotic arm adapted to support and move a tool and a second robotic arm adapted to support and move a camera configured to capture an image of a camera field of view. The system further comprises an input device, a display, and a processor. The processor is configured to display a first synthetic image including a first synthetic image of the tool. The first synthetic image of the tool includes a portion of the tool outside of the camera field of view. The processor is also configured to receive a user input at the input device and responsive to the user input, change the display of the first synthetic image to a display of a second synthetic image including a second synthetic image of the tool that is different from the first synthetic image of the tool.

Abstract: A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system possibly comprising a plurality of cameras that image the user's eye and glints for thereon and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of cornea of the user's eye using data derived from the glint images. The display system may use spherical and aspheric cornea models to estimate a location of the corneal center of the user's eye.

Abstract: A stereo imaging system comprises a depth sensing system, a stereo viewer, and a processor configured to receive captured stereo worksite images, compare a depth value to a predetermined threshold value, and generate stereo images for display by adjusting or not adjusting native disparities based on the comparison of the depth value to the threshold value.

Abstract: A system comprises a first robotic arm adapted to support and move a tool and a second robotic arm adapted to support and move a camera. The system also comprises an input device, a display, and a processor. The processor is configured to, in a first mode, command the first robotic arm to move the camera in response to a first input received from the input device to capture an image of the tool and present the image as a displayed image on the display. The processor is configured to, in a second mode, display a synthetic image of the first robotic arm in a boundary area around the captured image on the display, and in response to a second input, change a size of the boundary area relative a size of the displayed image.

Abstract: A robotic system includes a processor that is programmed to determine and cause work site measurements for user specified points in the work site to be graphically displayed in order to provide geometrically appropriate tool selection assistance to the user. The processor is also programmed to determine an optimal one of a plurality of tools of varying geometries for use at the work site and to cause graphical representations of at least the optimal tool to be displayed along with the work site measurements.