Abstract: Camera arrays for mediated-reality systems and associated methods and systems are disclosed herein. In some embodiments, a camera array includes a support structure having a center, and a depth sensor mounted to the support structure proximate to the center. The camera array can further include a plurality of cameras mounted to the support structure radially outward from the depth sensor, and a plurality of trackers mounted to the support structure radially outward from the cameras. The cameras are configured to capture image data of a scene, and the trackers are configured to capture positional data of a tool within the scene. The image data and the positional data can be processed to generate a virtual perspective of the scene including a graphical representation of the tool at the determined position.

Abstract: A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

Abstract: A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

Abstract: A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

Abstract: A method assigns weights to physical imager pixels in order to generate photorealistic images for virtual perspectives in real-time. The imagers are arranged in three-dimensional space such that they sparsely sample the light field within a scene of interest. This scene is defined by the overlapping fields of view of all the imagers or for subsets of imagers. The weights assigned to imager pixels are calculated based on the relative poses of the virtual perspective and physical imagers, properties of the scene geometry, and error associated with the measurement of geometry. This method is particularly useful for accurately rendering numerous synthesized perspectives within a digitized scene in real-time in order to create immersive, three-dimensional experiences for applications such as performing surgery, infrastructure inspection, or remote collaboration.

Abstract: A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

Abstract: A method assigns weights to physical imager pixels in order to generate photorealistic images for virtual perspectives in real-time. The imagers are arranged in three-dimensional space such that they sparsely sample the light field within a scene of interest. This scene is defined by the overlapping fields of view of all the imagers or for subsets of imagers. The weights assigned to imager pixels are calculated based on the relative poses of the virtual perspective and physical imagers, properties of the scene geometry, and error associated with the measurement of geometry. This method is particularly useful for accurately rendering numerous synthesized perspectives within a digitized scene in real-time in order to create immersive, three-dimensional experiences for applications such as performing surgery, infrastructure inspection, or remote collaboration.

Abstract: A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

Abstract: A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

Abstract: Camera arrays for mediated-reality systems and associated methods and systems are disclosed herein. In some embodiments, a camera array includes a plurality of hexagonal cells arranged in a honeycomb pattern, and a plurality of cameras mounted within each of the plurality of hexagonal cells. The plurality of cameras can include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

Abstract: A method assigns weights to physical imager pixels in order to generate photorealistic images for virtual perspectives in real-time. The imagers are arranged in three-dimensional space such that they sparsely sample the light field within a scene of interest. This scene is defined by the overlapping fields of view of all the imagers or for subsets of imagers. The weights assigned to imager pixels are calculated based on the relative poses of the virtual perspective and physical imagers, properties of the scene geometry, and error associated with the measurement of geometry. This method is particularly useful for accurately rendering numerous synthesized perspectives within a digitized scene in real-time in order to create immersive, three-dimensional experiences for applications such as performing surgery, infrastructure inspection, or remote collaboration.