Abstract: Methods for selecting pairs of cameras in a multicamera array configured to image a scene to generate stereoscopic views, and associated systems and devices, are disclosed herein. In some embodiments, a representative method includes selecting a pair of cameras in the multicamera array from which to generate a stereoscopic view, and receiving images of the scene from the selected pair of the cameras. The method further includes cropping the images based on a known calibration of the cameras and a desired disparity, and displaying the cropped images on a display device to generate the stereoscopic view. The desired disparity can correspond to an interpupillary distance of a user.

Abstract: Methods of determining the depth of a scene and associated systems are disclosed herein. In some embodiments, a method can include augmenting depth data of a scene captured with a depth sensor with depth data from one or more images of the scene. For example, the method can include capturing image data of the scene with a plurality of cameras. The method can further include generating a point cloud representative of the scene based on the depth data from the depth sensor and identifying a missing region of the point cloud, such as a region occluded from the view of the depth sensor. The method can then include generating depth data for the missing region based on the image data. Finally, the depth data for the missing region can be merged with the depth data from the depth sensor to generate a merged point cloud representative of the scene.

Abstract: A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

Abstract: Camera arrays for mediated-reality systems and associated methods and systems are disclosed herein. In some embodiments, a camera array includes a support structure having a center, and a depth sensor mounted to the support structure proximate to the center. The camera array can further include a plurality of cameras mounted to the support structure radially outward from the depth sensor, and a plurality of trackers mounted to the support structure radially outward from the cameras. The cameras are configured to capture image data of a scene, and the trackers are configured to capture positional data of a tool within the scene. The image data and the positional data can be processed to generate a virtual perspective of the scene including a graphical representation of the tool at the determined position.

Abstract: Endoscopic imaging systems are disclosed herein. In some embodiments, an endoscopic imaging system can include an endoscope having a distal tip with a distal face configured to face a scene, such as a portion of a body cavity of a patient. The endoscope can further include first cameras for capturing first image data of the scene, a projector for projecting a light pattern into the scene, and second cameras for capturing second image data of the scene including the light pattern. A processor can be communicatively coupled to the first and second cameras for receiving the first and second image data. The processor can process the first and second image data to generate an image of the scene at the perspective of a virtual camera, and can vary the perspective, the aperture, the focus plane, and/or another parameter of the virtual camera without requiring the endoscope to be physically moved.

Abstract: Tracking constellations for use with surgical instruments, and associated systems and methods, are disclosed herein. In some embodiments, a tracking constellation includes (i) a support, (ii) a plurality of first standoffs extending from the support to a first height, and (iii) a plurality of second standoffs extending from the support to a second height different than the first height. The tracking constellation can further include a plurality of markers mounted to corresponding ones of the first standoffs or the second standoffs. The markers can lay in a common plane. The support can extend at a generally orthogonal angle to an instrument when the tracking constellation is coupled to the instrument such that the common plane is angled relative to the instrument.

Abstract: Systems, devices, and methods for collection, editing, and playback of data collected from an imaging system for generating a virtual perspective of a scene are disclosed. In one example perspective, an imaging system includes a camera array configured to capture multiple images of a scene. Each of the multiple images includes color data represented in a Bayer pattern that includes a blue channel, two green channels, and a red channel. The system also includes an image processing device configured to receive the multiple images captured by the camera array, split each of the multiple images represented in the Bayer pattern into four individual color planes, form at least one set of data by combining the four individual color planes of the multiple images, and compress the at least one set of data.

Abstract: Methods and systems for intraoperatively determining alignment parameters of a spine during a spinal surgical procedure are disclosed herein. In some embodiments, a method of intraoperatively determining an alignment parameter of a spine during a surgical procedure includes receiving initial image data of the spine including multiple vertebrae and identifying a geometric feature associated with each vertebra in the initial image data. The geometric features each have a pose in the initial image data and characterize a three-dimensional (3D) shape of the associated vertebra. The method further comprises receiving intraoperative image data of the spine and registering the initial image data to the intraoperative image data. The method can then update the pose of each geometric feature based on the registration and the intraoperative image data, and determine the alignment parameter based on the updated poses of the geometric features associated with two or more of the vertebrae.

Abstract: A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

Abstract: A method assigns weights to physical imager pixels in order to generate photorealistic images for virtual perspectives in real-time. The imagers are arranged in three-dimensional space such that they sparsely sample the light field within a scene of interest. This scene is defined by the overlapping fields of view of all the imagers or for subsets of imagers. The weights assigned to imager pixels are calculated based on the relative poses of the virtual perspective and physical imagers, properties of the scene geometry, and error associated with the measurement of geometry. This method is particularly useful for accurately rendering numerous synthesized perspectives within a digitized scene in real-time in order to create immersive, three-dimensional experiences for applications such as performing surgery, infrastructure inspection, or remote collaboration.

Abstract: Methods of determining the depth of a scene and associated systems are disclosed herein. In some embodiments, a method can include augmenting depth data of a scene captured with a depth sensor with depth data from one or more images of the scene. For example, the method can include capturing image data of the scene with a plurality of cameras. The method can further include generating a point cloud representative of the scene based on the depth data from the depth sensor and identifying a missing region of the point cloud, such as a region occluded from the view of the depth sensor. The method can then include generating depth data for the missing region based on the image data. Finally, the depth data for the missing region can be merged with the depth data from the depth sensor to generate a merged point cloud representative of the scene.

Abstract: A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

Abstract: Camera arrays for mediated-reality systems and associated methods and systems are disclosed herein. In some embodiments, a camera array includes a support structure having a center, and a depth sensor mounted to the support structure proximate to the center. The camera array can further include a plurality of cameras mounted to the support structure radially outward from the depth sensor, and a plurality of trackers mounted to the support structure radially outward from the cameras. The cameras are configured to capture image data of a scene, and the trackers are configured to capture positional data of a tool within the scene. The image data and the positional data can be processed to generate a virtual perspective of the scene including a graphical representation of the tool at the determined position.

Abstract: A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

Abstract: Mediated-reality imaging systems, methods, and devices are disclosed herein. In some embodiments, an imaging system includes (i) a camera array configured to capture intraoperative image data of a surgical scene in substantially real-time and (ii) a processing device communicatively coupled to the camera array. The processing device can be configured to synthesize a three-dimensional (3D) image corresponding to a virtual perspective of the scene based on the intraoperative image data from the cameras. The imaging system is further configured to receive and/or store preoperative image data, such as medical scan data corresponding to a portion of a patient in the scene. The processing device can register the preoperative image data to the intraoperative image data, and overlay the registered preoperative image data over the corresponding portion of the 3D image of the scene to present a mediated-reality view.

Abstract: A method assigns weights to physical imager pixels in order to generate photorealistic images for virtual perspectives in real-time. The imagers are arranged in three-dimensional space such that they sparsely sample the light field within a scene of interest. This scene is defined by the overlapping fields of view of all the imagers or for subsets of imagers. The weights assigned to imager pixels are calculated based on the relative poses of the virtual perspective and physical imagers, properties of the scene geometry, and error associated with the measurement of geometry. This method is particularly useful for accurately rendering numerous synthesized perspectives within a digitized scene in real-time in order to create immersive, three-dimensional experiences for applications such as performing surgery, infrastructure inspection, or remote collaboration.

Abstract: Camera arrays for mediated-reality systems and associated methods and systems are disclosed herein. In some embodiments, a camera array includes a support structure having a center, and a depth sensor mounted to the support structure proximate to the center. The camera array can further include a plurality of cameras mounted to the support structure radially outward from the depth sensor, and a plurality of trackers mounted to the support structure radially outward from the cameras. The cameras are configured to capture image data of a scene, and the trackers are configured to capture positional data of a tool within the scene. The image data and the positional data can be processed to generate a virtual perspective of the scene including a graphical representation of the tool at the determined position.

Abstract: A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

Abstract: A method assigns weights to physical imager pixels in order to generate photorealistic images for virtual perspectives in real-time. The imagers are arranged in three-dimensional space such that they sparsely sample the light field within a scene of interest. This scene is defined by the overlapping fields of view of all the imagers or for subsets of imagers. The weights assigned to imager pixels are calculated based on the relative poses of the virtual perspective and physical imagers, properties of the scene geometry, and error associated with the measurement of geometry. This method is particularly useful for accurately rendering numerous synthesized perspectives within a digitized scene in real-time in order to create immersive, three-dimensional experiences for applications such as performing surgery, infrastructure inspection, or remote collaboration.