Abstract: A system and process for generating a panoramic video. Essentially, the panoramic video is created by first acquiring multiple videos of the scene being depicted. Preferably, these videos collectively depict a full 360 degree view of the surrounding scene and are captured using a multiple camera rig. The acquisition phase also includes a calibration procedure that provides information about the camera rig used to capture the videos that is used in the next phase for creating the panoramic video. This next phase, which is referred to as the authoring phase, involves mosaicing or stitching individual frames of the videos, which were captured at approximately the same moment in time, to form each frame of the panoramic video. A series of texture maps are then constructed for each frame of the panoramic video. Each texture map coincides with a portion of a prescribed environment model of the scene.

Abstract: A system and method for adjusting exposure in a mosaiced or stitched image. A stitched composite image is typically represented by a set of images and a set of associated transformations. Each transformation corresponds to one image in the input image sequence and represents the mapping between image pixels in each image and a three-dimensional coordinate system. Every triplet of images in the mosaiced image, having a first, center and third image, is input into the system and method according to the present invention. Before exposure adjustment can be performed, the regions of overlap between the input images in the mosaiced image are calculated. Once the areas of overlap associated with the first and third images are found, the areas of overlap associated with these images are warped to the coordinate frame of the center image.

Abstract: A system and process for generating a panoramic video. Essentially, the panoramic video is created by first acquiring multiple videos of the scene being depicted. Preferably, these videos collectively depict a full 360 degree view of the surrounding scene and are captured using a multiple camera rig. The acquisition phase also includes a calibration procedure that provides information about the camera rig used to capture the videos that is used in the next phase for creating the panoramic video. This next phase, which is referred to as the authoring phase, involves mosaicing or stitching individual frames of the videos, which were captured at approximately the same moment in time, to form each frame of the panoramic video. A series of texture maps are then constructed for each frame of the panoramic video. Each texture map coincides with a portion of a prescribed environment model of the scene.

Abstract: A system and process for deghosting mosaiced images created by stitching together images of a scene captured from different viewpoints is presented. When images are mosaiced, which were captured by different cameras at different viewpoints, the possibility of localized double images of objects exists. Essentially, this double imaging or ghosting will occur if an object in the scene is close in to the cameras capturing the images. However, this localized ghosting can be compensated for by estimating the amount of local mis-registration and then locally warping each image in the mosaiced image to reduce any ghosting.

Abstract: A system and method for manipulating a set of images of a static scene captured at different exposures (i.e., “bracketed” images) to yield a composite image with improved uniformity in exposure and tone. In general, the aforementioned goal can be achieved by analyzing a set of bracketed images using a multi-dimensional histogram and merging the images via an approach that projects pixels onto a curve that fits the data. However, it has been found that the desired composite image can be produced in a simpler manner by summing the pixel brightness levels across the multiple images, followed by an equalization process. One possible equalization process involves simply averaging the summed pixel brightness values by dividing the summed value of each pixel set (i.e., groups of corresponding pixels from the bracketed images) by the number of bracketed images. An even better result can be achieved using a histogram equalization process.

Abstract: A system and method for deghosting mosaics provides a novel multiperspective plane sweep approach for generating an image mosaic from a sequence of still images, video images, scanned photographic images, computer generated images, etc. This multiperspective plane sweep approach uses virtual camera positions to compute depth maps for columns of overlapping pixels in adjacent images. Object distortions and ghosting caused by image parallax when generating the image mosaics are then minimized by blending pixel colors, or grey values, for each computed depth to create a common composite area for each of the overlapping images. Further, the multiperspective plane sweep approach described herein is both computationally efficient, and applicable to both the case of limited overlap between the images used for creating the image mosaics, and to the case of extensive or increased image overlap.

Abstract: Automated layer extraction from 2D images making up a 3D scene, and automated image pixel assignment to layers, to provide for scene modeling, is disclosed. In one embodiment, a computer-implemented method determines a number of planes, or layers, and assigns pixels to the planes. The method can determine the number of planes by first determining the high-entropy pixels of the images, and then determining a 1-plane through a predetermined n-plane estimation, such as via a robust estimation, and a most likely x-plane estimation, where x is between 1 and n, such as via a Bayesian approach. Furthermore, the method can assign pixels via an iterative EM approach based on classifying criteria.

Abstract: A system and method for improving the uniformity in exposure and tone of a digital image using a locally adapted histogram equalization approach. This approach involves first segmenting the digital image into a plurality of image patches. For each of these patches, a pixel brightness level histogram is created. The histogram for each patch is then optionally averaged with the histograms associated with a prescribed number of neighboring image patches. A normalized cumulative distribution function is generated for each patch based on the associated averaged histogram. This normalized-cumulative distribution function identifies a respective new pixel brightness level for each of the original pixel brightness levels. For each of the original pixel brightness levels, the 1s associated new pixel brightness levels from one or more of the image patches are blended. Preferably, this blending is accomplished using either a bilinear or biquadratic interpolator function.

Abstract: A system and process for computing a 3D reconstruction of a scene using multiperspective panoramas. The reconstruction can be generated using a cylindrical sweeping approach, or under some conditions, traditional stereo matching algorithms. The cylindrical sweeping process involves projecting each pixel of the multiperspective panoramas onto each of a series of cylindrical surfaces of progressively increasing radii. For each pixel location on each cylindrical surface, a fitness metric is computed for all the pixels projected thereon to provide an indication of how closely a prescribed characteristic of the projected pixels matches. Then, for each respective group of corresponding pixel locations of the cylindrical surfaces, it is determined which location has a fitness metric that indicates the prescribed characteristic of the projected pixels matches more closely than the rest.

Abstract: A system and process for generating a new video sequence from frames taken from an input video clip. Generally, this involves computing a similarity value between each of the frames of the input video clip and each of the other frames. For each frame, the similarity values associated therewith are analyzed to identify potentially acceptable transitions between it and the remaining frames. A transition is considered acceptable if it would appear smooth to a person viewing a video containing the frames, or at least if the transition is one of the best available. A new video sequence is then synthesized using the identified transitions to specify an order in which the frames associated with these transitions are to be played. Finally, the new video sequence is rendered by playing the frames of the input video clip in the order specified in the synthesizing procedure.

Abstract: A system and process for generating a 3D video animation of an object referred to as a 3D Video Texture is presented. The 3D Video Texture is constructed by first simultaneously videotaping an object from two or more different cameras positioned at different locations. Video from, one of the cameras is used to extract, analyze and synthesize a video sprite of the object of interest. In addition, the first, contemporaneous, frames captured by at least two of the cameras are used to estimate a 3D depth map of the scene. The background of the scene contained within the depth map is then masked out, and a clear shot of the scene background taken before filming of the object began, leaving just the object. To generate each new frame in the 3D video animation, the extracted region making up a “frame” of the video sprite is mapped onto the previously generated 3D surface.

Abstract: A system and process for generating a video animation from the frames of a video sprite with user-controlled motion is presented. An object is extracted from the frames of an input video and processed to generate a new video sequence or video sprite of that object. In addition, the translation velocity of the object for each frame is computed and associated with each frame in the newly generated video sprite. The system user causes a desired path to be generated for the object featured in the video sprite to follow in the video animation. Frames of the video sprite showing the object of interest are selected and inserted in a background image, or frame of a background video, along the prescribed path. The video sprite frames are selected by comparing a last-selected frame to the other video sprite frames, and selecting a video sprite frame that is identified in the comparison as corresponding to an acceptable transition from the last-selected frame.

Abstract: The primary components of the panoramic video viewer include a decoder module. The purpose of the decoder module is to input incoming encoded panoramic video data and to output a decoded version thereof. The incoming data may be provided over a network and originate from a server, or it may simply be read from a storage media, such as a hard drive, CD or DVD. Once decoded, the data associated with each video frame is preferably stored in a storage module and made available to a 3D rendering module. The 3D rendering module is essentially a texture mapper that takes the frame data and maps the desired views onto a prescribed environment model. The output of the 3D rendering module is provided to a display module where the panoramic video is viewed by a user of the system. Typically, the user will be viewing just a portion of the scene depicted in the panoramic video at any one time, and will be able to control what portion is viewed.

Abstract: A system and process for computing motion or depth estimates from multiple images. In general terms this is accomplished by associating a depth or motion map with each input image (or some subset of the images equal or greater than two), rather that computing a single map for all the images. In addition, consistency between the estimates associated with different images is ensured. More particularly, this involves minimizing a three-part cost function, which consists of an intensity (or color) compatibility constraint, a motion/depth compatibility constraint, and a flow smoothness constraint. In addition, a visibility term is added to the intensity (or color) compatibility and motion/depth compatibility constraints to prevent the matching of pixels into areas that are occluded. In operation, the cost function is computed in two phases. During an initializing phase, the motion or depth for each image being examined are estimated independently.

Abstract: A system and method for inverse texture mapping in which given a 3D model and several images from different viewpoints, a texture map is extracted for each planar surface in the 3D model. The system and method employs a unique weighted pyramid feathering scheme for blending multiple images to form the texture map, even where the images are taken from different viewpoints, at different scales, and with different exposures. This scheme also makes it possible to blend images with cut-out regions which may be present due to occlusions or moving objects. It further advantageously employs weight maps to improve the quality of the blended image.

Abstract: A system and method for extracting structure from stereo that represents the scene as a collection of planar layers. Each layer optimally has an explicit 3D plane equation, a colored image with per-pixel opacity, and a per-pixel depth value relative to the plane. Initial estimates of the layers are recovered using techniques from parametric motion estimation. The combination of a global model (the plane) with a local correction to it (the per-pixel relative depth value) imposes enough local consistency to allow the recovery of shape in both textured and untextured regions.

Abstract: A system and method for extracting structure from stereo that represents the scene as a collection of planar layers. Each layer optimally has an explicit 3D plane equation, a colored image with per-pixel opacity, and a per-pixel depth value relative to the plane. Initial estimates of the layers are made and then refined using a re-synthesis step which takes into account both occlusions and mixed pixels. Reasoning about these effects allows the recovery of depth and color information with high accuracy, even in partially occluded regions. Moreover, the combination of a global model (the plane) with a local correction to it (the per-pixel relative depth value) imposes enough local consistency to allow the recovery of shape in both textured and untextured regions.

Abstract: An interactive system and process for constructing a model of a 3D scene from a panoramic view of the scene. In the constructed model, the 3D scene is represented by sets of connected planes. The modeling begins by providing the user with a display of an image of the panoramic view. The user is then required to specify information concerning certain geometric features of the scene. A computer program recovers a camera orientation matrix of the panoramic view based on the features specified by the user. Plane normals and line directions for planes in the 3D scene are estimated using this matrix as well as the user-specified information. A camera translation is also recovered, as are plane distances and vertex point locations for planes in the 3D scene, using the user-supplied information, camera orientation matrix, and the estimated plane normals and line directions.

Abstract: A system and method for creating weight maps capable of indicating how much each pixel in an image should contribute to a blended image. One such map is a view-dependent weight map created by inputting an image that has been characterized as a collection of regions. A 2D perspective transform is computed for each region that is to be part of the weight map. The transforms are used to warp the associated regions to prescribed coordinates to create a warped image. Once the warped image is created, a Jacobian matrix is computed for each pixel. The determinant of each Jacobian matrix is then computed to establish a weight factor for that pixel. The weight map for the inputted image is created using these computed determinants. Another advantageous weight map is a combination weight map. The process for creating type of weight map is identical to the view-dependant map up to the point the warped image has been created.

Abstract: A system and process for refining a model of a 3D scene using one or more panoramic views of the scene. An image of a panoramic view is displayed on a screen and a previously constructed model of the 3D scene is projected onto the screen image. Once the model is projected, any portion thereof that is not aligned with its corresponding feature in the screen image is moved so as to be in alignment. Plane normals and line directions for previously modeled planes in the 3D scene are then estimated using, inter alia, the newly aligned lines of the previous model. There may also be new, unmodeled features appearing in the screen image. These features can also be modeled, if desired. Next, plane distances and vertex point locations of each plane in the 3D scene that is to be modeled are estimated.