Abstract: A system comprises: a robotic arm operatively coupleable to a tool comprising a working end; and an input device communicatively coupled to the robotic arm. The input device is manipulatable by an operator. The system further comprises a processor configured to cause an image of a work site, captured by an image capture device from a perspective of an image reference frame, to be displayed on a display. The image of the work site includes an image of the working end of the tool. The processor is further configured to determine a position of the working end of the tool in the image of the work site and render a tool information overlay at the position of the working end of the tool in the image of the work site. The tool information overlay visually indicates an identity of the input device.

Abstract: A system may comprise a tool including at least one reference feature. a processor, and a memory having computer readable instructions stored thereon. The computer readable instructions, when executed by the processor, may cause the system to receive image data including an image of the tool and the at least one reference feature, determine a pose of the tool from the image data, and modify the image data to visually decrement a portion of the image data corresponding to the at least one reference feature.

Abstract: An example method includes causing an illuminator to simultaneously illuminate tissue with a first visible color illumination component, a second visible color illumination component, and a fluorescence excitation illumination component; and causing a display unit to display, based on the illumination, a display scene that includes: a reduced color scene component corresponding to the first and second visible color illumination components, and a highlighted scene component corresponding to the fluorescence excitation illumination component.

Abstract: The systems and methods described can include approaches to calibrate head-mounted displays for improved viewing experiences. Some methods include receiving data of a first target image associated with an undeformed state of a first eyepiece of a head-mounted display device; receiving data of a first captured image associated with deformed state of the first eyepiece of the head-mounted display device; determining a first transformation that maps the first captured image to the image; and applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

Abstract: A flexible display panel includes a flexible display body and a supporting structure disposed on a side of the flexible display body. The supporting structure includes a plurality of supporting layers disposed on a stack and a flexible layer disposed between each two adjacent supporting layers. An angle between at least one supporting layer of the plurality of supporting layers and a plane where a display surface of the flexible display body is located is greater than 0° and less than 90°.

Abstract: A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system possibly comprising a plurality of cameras that image the user's eye and glints thereon and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of cornea of the user's eye using data derived from the glint images. The display system may use spherical and aspheric cornea models to estimate a location of the corneal center of the user's eye.

Abstract: A system comprises: a robotic arm operatively coupleable to a tool comprising a working end; and an input device communicatively coupled to the robotic arm. The input device is manipulatable by an operator. The system further comprises a processor configured to cause an image of a work site, captured by an image capture device from a perspective of an image reference frame, to be displayed on a display. The image of the work site includes an image of the working end of the tool. The processor is further configured to determine a position of the working end of the tool in the image of the work site and render a tool information overlay at the position of the working end of the tool in the image of the work site. The tool information overlay visually indicates an identity of the input device.

Abstract: A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system possibly comprising a plurality of cameras that image the user's eye and glints thereon, and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of cornea of the user's eye using data derived from the glint images. The display system may use spherical and aspheric cornea models to estimate a location of the corneal center of the user's eye.

Abstract: A medical robotic system includes a viewer for displaying an image of a work site, a gaze tracker for tracking a gaze point of a user on the viewer, and a processor. The processor is configured to: draw an area or volume defining shape, overlaid on the image of the work site, in a position determined by the gaze tracker; assign a fixed virtual constraint to the shape and constrain movement of a robotic tool according to the fixed virtual constraint; receive a user selected action command selecting an image of patient anatomy; and superimpose the selected image of the patient anatomy over the image of the work site within the shape. The selected image of the patient anatomy is registered to the image of the work site.

Abstract: A robotic system may comprise a first robotic arm operatively coupleable to a first tool. The first tool has a first working end. The system may also comprise an image capture device, a display, and a processor. The processor may be configured to cause an image of a work site, which was captured by the image capture device from a perspective of an image reference frame, to be displayed on the display. The image of the work site includes an image of the first working end of the first tool. The processor may also determine a position of the first working end of the first tool in the image of the work site and render a tool information overlay at the position of the first working end of the first tool in the image of the work site. The tool information overlay visually indicates state information for the first tool. The processor may also change the tool information overlay while the first tool is in a first operational state by changing a brightness of the tool information overlay.

Abstract: The systems and methods described can include approaches to calibrate head-mounted displays for improved viewing experiences. Some methods include receiving data of a first target image associated with an undeformed state of a first eyepiece of a head-mounted display device; receiving data of a first captured image associated with deformed state of the first eyepiece of the head-mounted display device; determining a first transformation that maps the first captured image to the image; and applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

Abstract: An exemplary method includes receiving a frame of raw pixels captured by an image capture sensor, a particular location in the frame of raw pixels including a captured color pixel value of a first color component of a first color space; generating, for the particular location in the frame of raw pixels, a vertical color difference signal and a horizontal color difference signal using information from the frame of raw pixels; determining that the particular location is missing a color pixel value of a second color component of the first color space; generating a reconstructed color pixel value of the second color component based on the vertical color difference signal and the horizontal color difference signal; and transforming the vertical color difference signal and the horizontal color difference signal into chrominance pixel values in a second color space that is different from the first color space.

Abstract: A computing platform (10), a method, and an apparatus (20) for spiking neural network (SNN) learning and simulation are provided. The computing platform (10) includes a neuron dynamics simulation module (11), a neuron conversion module (12), an SNN construction and weight learning module (13), and a neural network level parameter and weight access module (14). The neuron dynamics simulation module (11) simulates changing features of neurons. The neuron conversion module (12) performs operations on a calculation graph. The SNN construction and weight learning module (13) updates and iterates connection weights. The neural network level parameter and weight access module (14) stores overall network detail parameters.

Abstract: An example method includes causing an illuminator to simultaneously illuminate tissue with a first visible color illumination component, a second visible color illumination component, and a fluorescence excitation illumination component; and causing a display unit to display, based on the illumination, a display scene that includes: a reduced color scene component corresponding to the first and second visible color illumination components, and a highlighted scene component corresponding to the fluorescence excitation illumination component.

Abstract: A calibration assembly may be used with an endoscopic imaging system having a field of view. The calibration assembly may comprise an interface configured for constraining engagement with the endoscopic imaging system and a target coupled with the interface so as to be within the field of view. The target may include a plurality of calibration features. The calibration assembly may be reconfigurable from a first spatial arrangement between the target and the endoscopic imaging system to a second spatial arrangement between the target and the endoscopic imaging system. The calibration assembly may also comprise a processor configured to obtain, from the endoscopic imaging system, first and second images of at least some of the calibration features of the target at the first and second spatial arrangements. The processor may also generate calibration data for the endoscopic imaging system using the first and second images.

Abstract: The systems and methods described can include approaches to calibrate head-mounted displays for improved viewing experiences. Some methods include receiving data of a first target image associated with an undeformed state of a first eyepiece of a head-mounted display device; receiving data of a first captured image associated with deformed state of the first eyepiece of the head-mounted display device; determining a first transformation that maps the first captured image to the image; and applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

Abstract: A target tracking method and a target tracking system of a spiking neural network based on an event camera are provided. The method includes: acquiring a data stream of asynchronous events in a high dynamic scene of a target by an event camera as input data; dividing the data stream of the asynchronous events into synchronous event frames with millisecond time resolution; training a twin network based on a spiking neural network by a gradient substitution algorithm with a target image as a template image and a complete image as a searched image; and tracking the target by a trained twin network with interpolating a result of feature mapping to up-sample and obtaining the position of the target in an original image. The twin network includes a feature extractor and a cross-correlation calculator.

Abstract: An exemplary method includes a demosaic module receiving a frame of raw pixels having values in a first color space, generating vertical and horizontal color difference signals using information from the frame of raw pixels, generating reconstructed color pixel values for raw pixels that are missing captured color pixel values of a first color component of the first color space, transforming the captured and reconstructed color pixel values of the first color component to brightness pixel values in a second color space, and transforming the vertical and horizontal color difference signals into chrominance pixel values in the second color space.

Abstract: A system may comprise an image capture device to capture an image of a work site. The system may also comprise a processor to determine whether a tool disposed at the work site is energized and determine a first area of the captured image of the work site, including or adjacent to an image of a portion of the tool in the captured image. The processor may also determine a second area of the captured image including a remainder of the captured image that does not include the first area. Conditioned upon determining that the tool is energized, at least one of the first area and the second area of the captured image of the work site may be processed to indicate that the tool in the first area is being energized. The image of the work site with the first area and the second area may be displayed.

Abstract: An example method includes receiving a single frame comprising a first set of pixel data that includes a fluorescence scene component and a second set of pixel data that includes a combination of a visible color component scene and the fluorescence scene component; and generating, based on the first set of pixel data that includes the fluorescence scene component and the second set of pixel data that includes the combination of the visible color component scene and the fluorescence scene component, a display scene.