Abstract: Endoscopic imaging systems are disclosed herein. In some embodiments, an endoscopic imaging system can include an endoscope having a distal tip with a distal face configured to face a scene, such as a portion of a body cavity of a patient. The endoscope can further include first cameras for capturing first image data of the scene, a projector for projecting a light pattern into the scene, and second cameras for capturing second image data of the scene including the light pattern. A processor can be communicatively coupled to the first and second cameras for receiving the first and second image data. The processor can process the first and second image data to generate an image of the scene at the perspective of a virtual camera, and can vary the perspective, the aperture, the focus plane, and/or another parameter of the virtual camera without requiring the endoscope to be physically moved.

Abstract: Systems, devices, and methods for collection, editing, and playback of data collected from an imaging system for generating a virtual perspective of a scene are disclosed. In one example perspective, an imaging system includes a camera array configured to capture multiple images of a scene. Each of the multiple images includes color data represented in a Bayer pattern that includes a blue channel, two green channels, and a red channel. The system also includes an image processing device configured to receive the multiple images captured by the camera array, split each of the multiple images represented in the Bayer pattern into four individual color planes, form at least one set of data by combining the four individual color planes of the multiple images, and compress the at least one set of data.

Abstract: Methods and systems for intraoperatively determining alignment parameters of a spine during a spinal surgical procedure are disclosed herein. In some embodiments, a method of intraoperatively determining an alignment parameter of a spine during a surgical procedure includes receiving initial image data of the spine including multiple vertebrae and identifying a geometric feature associated with each vertebra in the initial image data. The geometric features each have a pose in the initial image data and characterize a three-dimensional (3D) shape of the associated vertebra. The method further comprises receiving intraoperative image data of the spine and registering the initial image data to the intraoperative image data. The method can then update the pose of each geometric feature based on the registration and the intraoperative image data, and determine the alignment parameter based on the updated poses of the geometric features associated with two or more of the vertebrae.

Abstract: Methods and systems for calibrating an imaging system having a plurality of sensors are disclosed herein. In some embodiments, a method includes initially calibrating the imaging system, operating the imaging system during an imaging procedure, and then updating the calibration during the imaging procedure to account for degradation of the initial calibration due to environmental factors, such as heat. The method of updating the calibration can include capturing image data of a rigid body having a known geometry with the sensors and determining that the calibration has drifted for a problematic one of the sensors based on the captured image data. After determining the problematic one of the sensors, the method can include updating the calibration of the problematic one of the sensors based on the captured image data.

Abstract: Tracking constellations for use with surgical instruments, and associated systems and methods, are disclosed herein. In some embodiments, a tracking constellation includes (i) a support, (ii) a plurality of first standoffs extending from the support to a first height, and (iii) a plurality of second standoffs extending from the support to a second height different than the first height. The tracking constellation can further include a plurality of markers mounted to corresponding ones of the first standoffs or the second standoffs. The markers can lay in a common plane. The support can extend at a generally orthogonal angle to an instrument when the tracking constellation is coupled to the instrument such that the common plane is angled relative to the instrument.

Abstract: Methods and systems for calibrating an instrument, such as a surgical instrument, within an imaging system are disclosed herein. In some embodiments, a method includes capturing images of the instrument with a plurality of cameras of the imaging system and identifying common features of the instrument in the captured images. The method further includes generating a three-dimensional (3D) representation of the instrument based on the common features and determining a reference frame of the instrument based on the generated 3D representation of the instrument. A first transform is determined between the reference frame of the instrument and a reference frame of the cameras. Then, a second transform between the reference frame of the instrument and a reference frame of the tracking structure can be determined based on the first transform.

Abstract: Camera arrays for mediated-reality systems and associated methods and systems are disclosed herein. In some embodiments, a camera array includes a support structure having a center, and a depth sensor mounted to the support structure proximate to the center. The camera array can further include a plurality of cameras mounted to the support structure radially outward from the depth sensor, and a plurality of trackers mounted to the support structure radially outward from the cameras. The cameras are configured to capture image data of a scene, and the trackers are configured to capture positional data of a tool within the scene. The image data and the positional data can be processed to generate a virtual perspective of the scene including a graphical representation of the tool at the determined position.

Abstract: Methods for selecting pairs of cameras in a multicamera array configured to image a scene to generate stereoscopic views, and associated systems and devices, are disclosed herein. In some embodiments, a representative method includes selecting a pair of cameras in the multicamera array from which to generate a stereoscopic view, and receiving images of the scene from the selected pair of the cameras. The method further includes cropping the images based on a known calibration of the cameras and a desired disparity, and displaying the cropped images on a display device to generate the stereoscopic view. The desired disparity can correspond to an interpupillary distance of a user.

Abstract: A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

Abstract: Camera arrays for mediated-reality systems and associated methods and systems are disclosed herein. In some embodiments, a camera array includes a support structure having a center, and a depth sensor mounted to the support structure proximate to the center. The camera array can further include a plurality of cameras mounted to the support structure radially outward from the depth sensor, and a plurality of trackers mounted to the support structure radially outward from the cameras. The cameras are configured to capture image data of a scene, and the trackers are configured to capture positional data of a tool within the scene. The image data and the positional data can be processed to generate a virtual perspective of the scene including a graphical representation of the tool at the determined position.

Abstract: Endoscopic imaging systems are disclosed herein. In some embodiments, an endoscopic imaging system can include an endoscope having a distal tip with a distal face configured to face a scene, such as a portion of a body cavity of a patient. The endoscope can further include first cameras for capturing first image data of the scene, a projector for projecting a light pattern into the scene, and second cameras for capturing second image data of the scene including the light pattern. A processor can be communicatively coupled to the first and second cameras for receiving the first and second image data. The processor can process the first and second image data to generate an image of the scene at the perspective of a virtual camera, and can vary the perspective, the aperture, the focus plane, and/or another parameter of the virtual camera without requiring the endoscope to be physically moved.

Abstract: Tracking constellations for use with surgical instruments, and associated systems and methods, are disclosed herein. In some embodiments, a tracking constellation includes (i) a support, (ii) a plurality of first standoffs extending from the support to a first height, and (iii) a plurality of second standoffs extending from the support to a second height different than the first height. The tracking constellation can further include a plurality of markers mounted to corresponding ones of the first standoffs or the second standoffs. The markers can lay in a common plane. The support can extend at a generally orthogonal angle to an instrument when the tracking constellation is coupled to the instrument such that the common plane is angled relative to the instrument.

Abstract: Systems, devices, and methods for collection, editing, and playback of data collected from an imaging system for generating a virtual perspective of a scene are disclosed. In one example perspective, an imaging system includes a camera array configured to capture multiple images of a scene. Each of the multiple images includes color data represented in a Bayer pattern that includes a blue channel, two green channels, and a red channel. The system also includes an image processing device configured to receive the multiple images captured by the camera array, split each of the multiple images represented in the Bayer pattern into four individual color planes, form at least one set of data by combining the four individual color planes of the multiple images, and compress the at least one set of data.

Abstract: Methods and systems for intraoperatively determining alignment parameters of a spine during a spinal surgical procedure are disclosed herein. In some embodiments, a method of intraoperatively determining an alignment parameter of a spine during a surgical procedure includes receiving initial image data of the spine including multiple vertebrae and identifying a geometric feature associated with each vertebra in the initial image data. The geometric features each have a pose in the initial image data and characterize a three-dimensional (3D) shape of the associated vertebra. The method further comprises receiving intraoperative image data of the spine and registering the initial image data to the intraoperative image data. The method can then update the pose of each geometric feature based on the registration and the intraoperative image data, and determine the alignment parameter based on the updated poses of the geometric features associated with two or more of the vertebrae.

Abstract: A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

Abstract: A method assigns weights to physical imager pixels in order to generate photorealistic images for virtual perspectives in real-time. The imagers are arranged in three-dimensional space such that they sparsely sample the light field within a scene of interest. This scene is defined by the overlapping fields of view of all the imagers or for subsets of imagers. The weights assigned to imager pixels are calculated based on the relative poses of the virtual perspective and physical imagers, properties of the scene geometry, and error associated with the measurement of geometry. This method is particularly useful for accurately rendering numerous synthesized perspectives within a digitized scene in real-time in order to create immersive, three-dimensional experiences for applications such as performing surgery, infrastructure inspection, or remote collaboration.

Abstract: Methods of determining the depth of a scene and associated systems are disclosed herein. In some embodiments, a method can include augmenting depth data of a scene captured with a depth sensor with depth data from one or more images of the scene. For example, the method can include capturing image data of the scene with a plurality of cameras. The method can further include generating a point cloud representative of the scene based on the depth data from the depth sensor and identifying a missing region of the point cloud, such as a region occluded from the view of the depth sensor. The method can then include generating depth data for the missing region based on the image data. Finally, the depth data for the missing region can be merged with the depth data from the depth sensor to generate a merged point cloud representative of the scene.

Abstract: A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

Abstract: Camera arrays for mediated-reality systems and associated methods and systems are disclosed herein. In some embodiments, a camera array includes a support structure having a center, and a depth sensor mounted to the support structure proximate to the center. The camera array can further include a plurality of cameras mounted to the support structure radially outward from the depth sensor, and a plurality of trackers mounted to the support structure radially outward from the cameras. The cameras are configured to capture image data of a scene, and the trackers are configured to capture positional data of a tool within the scene. The image data and the positional data can be processed to generate a virtual perspective of the scene including a graphical representation of the tool at the determined position.

Abstract: A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.