Representation and coding of panoramic and omnidirectional images
A method for representing an omni directional-like image by capturing an omni-directional-like image and determining a representation format that is commonly applicable to any image acquisition system that is able to capture the omni-directional- like image. The captured omni-directional-like image is mapped onto the determined representation format.
This is a divisional of application Ser. No. 10/308,828, filed on Dec. 3, 2002, entitled “Representation and Coding of Panoramic and Omnidirectional Images,” and assigned to the corporate assignee of the present invention and incorporated herein by reference.
BACKGROUND OF THE INVENTION1. Field of the Invention
The present invention relates to the field of imaging, in particular the field of panoramic or omni-directional-like imaging.
2. Discussion of Related Art
It is well known in the art that panoramic or omni-directional-like images can be generated in a number ways. For example, such images can be generated by either rotating a single camera fixed at a given point in space followed by mosaicing the captured images, or by using a catadioptric camera equipped with a well known lens-mirror system. In each of these cases, the resulting image(s) format and organization will be specific to the particular acquisition system used. Hence, aside from image or video encoder performance that is dependent on the type of acquisition used, a separate decoder must be used for decoding each type of image. Additionally, if the encoder directly codes the images as acquired by the image acquisition system, or a modified version of them which is still dependent on the particular acquisition system, the burden of computing the inverse mapping necessary to generate and display perspective images from the decoded omnidirectional images falls on the decoder.
The dependency of the encoder performance on the various captured image formats, the dependency of the decoder on a particular acquisition system (and encoding format), and the associated computational burden placed on the decoder are among the major disadvantages of previous omnidirectional representation and coding approaches (e.g., the ParaMax360 system manufactured by Remote Reality, Inc. of Westborough, Mass.). Also, previous formats in which the panoramic and omnidirectional image data was generated in the art are not well-suited for image or video encoding using standard image or video codecs, given the nature of omni-directional-like images such as their shape, optical flow, and distortion of the image.
In the context of the need to store, process, transmit, and display panoramic or omni-directional-like images, new requirements have consequently arisen in terms of their representation, coding, transmission, rendering and display. New approaches are required to enable a generation of multimedia systems where both the video and audio components reflect recent advances in the capture processes of still images and video.
SUMMARY OF THE INVENTIONOne aspect of the present invention regards a method for determining a parameter of an image acquisition system from an omni-directional-like image by generating an omni-directional-like image via an image acquisition system and directly determining a parameter of the image acquisition system based on a geometrical characteristic of the omni-directional-like image.
A second aspect of the present invention regards a method for dynamically calibrating an image acquisition system from a plurality of frames of an omni-directional-like image. The method includes generating a first omni-directional-like image via an image acquisition system and generating first calibration information regarding the image acquisition system based on the first omni-directional-like image. Information is generated regarding a second omni-directional-like image generated by the image acquisition system, wherein the information is based on the first calibration information.
In a variation of the second aspect of the invention, the calibration information associated with the second omni-directional image can be generated by utilizing the image data from this image and the calibration information determined based on the previous omni-directional image. The process can be repeated for subsequent captured omni-directional images, where the calibration information of more than one image in the past can be used to determine the calibration information for the current image.
A third aspect of the present invention regards a method for representing an omni directional-like image by capturing an omni-directional-like image and determining a representation format that is commonly applicable to any image acquisition system that is able to capture the omni-directional-like image. The captured omni-directional-like image is mapped onto the determined representation format.
In a variation of the third aspect of the present invention, the omni-directional image(s) transformed into the representation format is (are) encoded using an image or video encoder. The omni-directional images encoded in this manner can be decoded and displayed using a common image or video decoder.
One or more of the above aspects of the present invention provides the advantage of presenting a universal mode of representing panoramic or omni-directional-like images independent of the type of acquisition system used to generate the image. The universal image format has a number of desirable properties that include a rectangular format and continuity of image features. Another advantage is that omni-directional images represented in the universal format can be efficiently encoded, transmitted, decoded, and displayed using standard image or video encoders and decoders. Also, rather than using a separate type of image or video decoder corresponding to the type of images acquired by a specific image acquisition system, a single common decoder can be used to decode and display omni-directional images utilizing the present invention. Similarly, as a consequence of decoding and rendering from the universal format of this invention, the processing load is further biased toward the encoder thus allowing a simpler and more efficient operation of the corresponding image or video decoder.
The present invention, together with attendant objects and advantages, will be best understood with reference to the detailed description below in connection with the attached drawings.
DESCRIPTION OF THE DRAWINGS
The general principles of the present invention can be best understood by first considering an ideal, single center of an image acquisition system, which can capture an entire sphere of view. As shown in
Since the preferred imaging device generates a planar image for viewing purposes, the points on the virtual sphere S consisting of the points P are preferably further projected onto a planar surface into a desired format. As it is well known, unavoidable problems related to distortion and sampling arise when the mapping from a sphere to a plane takes place. These problems are well known because they have been dealt with extensively in the field of cartography, where a number of mappings have been determined satisfying various requirements and presenting corresponding trade-offs in terms of distortion, shape and area preservation, etc. Aside from the necessity of mapping points on a sphere to a plane, the present invention may place additional constraints on the projection required, such as that the resulting mapped (or transformed) image in the universal format be rectangular and thus suitable for coding with a standard image or video coder, and also preserve as much as possible the continuity on the virtual sphere's surface.
The above described imaging process can be described mathematically. For example, a mapping τ is determined which relates the captured omnidirectional image E to a transformed image I having a rectangular format, all of whose pixels are utilized for the representation of the visual information contained in the captured omnidirectional image E. Thus, the mapping τ determines how to compute the value of a pixel in the universal format image I based on pixels values from the captured image E:
τ:EI (1)
The virtual sphere of view S, which can be associated with any single center of the image acquisition system, represents the common transfer entity which allows the overall mapping τ to be separated into two main parts. The first main part is acquisition system dependent, and relates the acquired omni-directional image data E to the corresponding virtual sphere of view S. The second main part is a common mapping which is independent of the particular image acquisition system used, and which transfers the visual information from the sphere of view S to a planar, rectangular universal image format I.
At a general description level, the required mappings that have to be determined to compute the overall mapping τ are as follows. The world points T are projected onto the surface of the virtual sphere S centered at the focal point O. A direct mapping of the virtual sphere of view S to its equatorial plane parallel to the image plane of the imaging device, such as a camera, is determined such that this projection results in the same image of the world points T as that obtained through the actual physical process. This mapping corresponds (through an equivalent model) to the process by which a panoramic or omni-directional-like image is captured by the image acquisition system. This step is dependent on the particular image acquisition system used for capturing panoramic or omni-directional-like images. The above described mapping determines the projection to the sphere S of the captured panoramic or omni-directional data in image E, and is denoted by τd:
τd:E→S.
Furthermore, for the projection of the visual information transferred to the virtual sphere S (through the mapping τd), to a well-specified planar rectangular image I, the mappings τ1 and τ2 must be determined. τ1 represents a mapping from the virtual sphere of view S centered at center of projection O, to the surface X of a well-determined solid having planar faces (polyhedra), with its center situated at O. A mapping τ2 is also computed, which projects the corresponding planar faces of the solid X to their corresponding orthogonal projections in the equatorial plane of the sphere S (parallel to the image plane of the projection image acquisition device). τ1 and τ2 are given below:
τ1:S→X,
τ2:X→I.
These computed mappings are then used to determine the overall mapping τ between the captured panoramic or omni-directional-like image E and the mapped image I which represents the universal format required, where τ uses interpolation of the pixel values in the captured omni-directional-like image E to determine pixel values of the final rectangular image I:
τ=(τ2oτ1) Oτd:E→I
Thus, from a practical point of view, for a sequential scan of the pixels of I, the value of each pixel in I is determined through the mapping τ (more precisely, its inverse), from the pixel values in the captured omni-directional-like image E. While the formation of the captured image E is acquisition system-dependent, the final image format I into which the visual information associated with S is transferred is not. This provides the advantage related to the formation of a rectangular image format which is suitable for encoding/decoding using standard image or video codecs. This also provides the advantages related to the decoding and rendering of the omni-directional-like images, and the distribution of processing load between the encoder and decoder. As a direct consequence of the universal image representation format used, a single decoder can be used to decode images captured with different image acquisition systems.
Once the mapping τd has been determined allowing image data in the captured omni-directional-like image E to be projected onto the sphere S, a mapping from the normalized unit sphere of view S to a common image format I needs to be determined (corresponding to the previously introduced composite mapping τ2oτ1), in order to represent captured omni-directional-like images in a common format which is independent of the image acquisition system and which has a number of well-determined properties. In this case, the sampled surface of the sphere S is projected onto a plane in a number of ways. A common characteristic of any such projection from the sphere onto a plane is that there are trade-offs between properties of the particular projection chosen, such as distortion, area preservation, and sphere sampling uniformity. While cartographic techniques of mapping the spherical surface of the Earth onto a plane are known, such techniques result in plane projections which are not amenable to image or video encoding using standard encoding schemes. Other known projection approaches use a projection from S to an intermediate solid inscribed in the sphere, such as projection to a cube or icosahedron, whose faces are then unwrapped in a plane. Aside from sampling non-uniformity (e.g., cube corners versus other areas), the unwrapping of these polyhedra in the plane does not directly result in a full rectangular image suitable for encoding using a standard image encoder, and result in a further increase in the discontinuity among the unwrapped faces in the plane which can affect the performance of an image or video encoder. The surface of the sphere can also be approximated by using meshes (e.g., triangular), and the resulting planar elements can be packed in an image. However, the packing of the surface elements in a rectangular image is not trivial, and can result in loss of continuity along edges of the elements (e.g., triangles), spaces between the packed elements, as well as incomplete packing in a rectangular format.
The present invention employs a projection from the sphere S into the universal image format I involving two phases. The first phase corresponds to the mapping τ1 discussed previously where the points on the sphere S are projected onto an intermediate solid (polyhedra) X. After this phase, the points on the intermediate surface X are projected into the final image format I, such as a rectangular format, per mapping τ2. In order to obtain a rectangular image format I, the intermediate polyhedra X is selected to be an octahedron, whose center is also situated at O. As shown in
As shown in
There are several possible ways to process the resulting two rectangular images. Firstly, the two images could be processed separately. Secondly, the two images can be packed together in a way that results in a rectangular image I, while preserving as much as possible the continuity that exists on the sphere S. One possible way to pack the images is shown in
The image I can be encoded using a standard image coder 112 such as JPEG. Similarly, a sequence of common format images {Ik} can be encoded using a video encoder 112 such as MPEG. Thus, a common decoder 114 can be used to decode omni-directional-like images originating from different acquisition systems and represented in the common image format I. The decoder 114 only has to texture-map the decoded universal-format image I to the solid X used for representation, which is an operation that can be very efficiently computed by rendering engines. Furthermore, the processing load asymmetry is shifted to the encoder 112 which will generate the images in the proposed common format. This is a desirable result given the much more stringent execution speed constraints that are placed on the operation of the decoder.
As shown in
An embodiment of the present invention uses the lens-mirror package manufactured by Remote Reality, Inc. under the trade name ParaMax 360 that is attached to a Sony DV -2000 video camera only for acquiring the raw omni-directional-like images. The raw omni-directional-like images are not coded or processed via software/algorithms available with the ParaMax 360 system. Instead, the captured images are further processed using the methods comprising the present invention. Note that the combination of the parabolic mirror 102 and telecentric lens 104 ensures a single center of projection. Furthermore, the parabolic mirror 102 has focal distance f with an axis of symmetry which coincides with the axis of an orthographic imaging device. The parabolic mirror 102 has a maximum vertical angle-of-view limit of α, above the horizontal plane that cuts the paraboloid to create the finite mirror. Also, given the orthographic projection from the surface of the paraboloid to the image plane, the distance of the image plane from the paraboloid is irrelevant. In addition, the image plane is perpendicular to the axis of symmetry of the paraboloid and passes through the focal point of the paraboloid. Preferably, an aspect ratio of 1, and no skew is present.
In order to ensure that the final representation of the captured image data is independent of the particular image acquisition system used to capture the visual information, a mapping must be determined which relates the particular image acquisition system employed to a common geometric model which will then permit a unified method of further projecting the visual information into the final format adopted. As previously presented, the present invention employs the normalized sphere of view S centered in the focal point of the parabolic mirror as an intermediate geometrical construct onto which the visual information is projected in a manner specific to the geometry of the image acquisition system 100. In the case of using a parabolic mirror 102 having a focal length f, the associated non-normalized virtual sphere S has a radius r =2*f. After the projection of the visual information contained in the captured omni-directional-like image E onto the virtual sphere S, the visual information on the surface of S is further projected into the final image representation format I in a manner that is independent of the acquisition system and depends only on the method by which the sphere S is projected into the chosen format (e.g., through an intermediate, planar-faced inscribed solid), and the requirements imposed on that format.
For the image acquisition system 100, shown in
The calibration includes two main phases after the image E has been captured by the image acquisition system 100 per step 202. During the first phase, the edge (or portions of it) separating the disc-shaped panoramic image from the unused pixel areas is detected per step 204. An edge detector (e.g., Sobel operator) is used for this purpose resulting in a binary image. The binary image W that results has pixels with value of “1” that represent a noisy set of pixels corresponding to areas of the edge of the circular acquired panoramic image. A priori knowledge about extremes of the position of this outer edge in the image can be used to prefilter the image W (and thus further reduce the number of candidate pixels with value “1”), but it is not necessary.
During the second phase of the calibration process, the retained noisy set of pixels (pixels having value “1” in image W) are fitted to a circle having radius r, and center (u′o,v′o) per step 206, where u′ and v′ represent the Cartesian system of coordinates in image W. Instead of using a non-robust least-square method in the original pixel space, a robust technique based on a parametric space operation and a majority voting technique is used as described hereafter. Taking the equation of a circle in polar coordinates:
u′=u′o+r cos θ
v′=v′o+r sin θ;
where u′o, v′o represent the coordinates of the center of the circle and r is its radius.
It follows that v′o=u′o tan θ+(v′−u′ tan θ). The parameters u′o, v′o representing the center of the circle can be determined using a majority voting technique. Thus, a two-dimensional array (accumulator) C(u′o, v′o) is set up, with initial values all zero. From the application of the edge detector to the image E, the gradient angle θ(u′,v′) is available for all retained pixels in the input image W. For each pixel in W, all the positions in the array C(u′o, v′o) are incremented along a line defined by the equation v′o=u′o tan θ+(v′−u′ tan θ). The maxima of the accumulator obtained after all pixels in W have been processed represent the center of the circle (u′o,v′o). The radius r is then found using the determined center of the circle. The values of the parameters in this case can be found per step 208 with subpixel precision by appropriately constructing (quantizing) the parameter space array C(u′o,v′o) and using the equation v′o=u′o tan θ+(v′−u′ tan θ).
The image of a sample parameter array C obtained after processing all the pixels in a sample image W, and containing the parameter lines given by the equation v′o=u′o tan θ+(v′−u′ tan θ) is shown in
Once the calibration process is complete, it is possible to determine the overall mapping τ between points J (which, with scaling, form the captured image E shown in
For an embodiment of the present invention, shown in
The determination of the overall mapping τ from the captured omni-directional-like image E to the universal image format I is detailed for an embodiment of the present invention shown in
Next, the system of coordinates {u′,v′} situated in the equator plane of the sphere S is introduced, whose axes coincide with {x,y} correspondingly, and which is centered at O (
Using homogeneous coordinates, the relation between the coordinates of points J(u′,v′) and P(x,y,z) is computed per the following relationships:
u′=u′(x,y,z)
v′=v′(x,y,z).
For the case of a parabolic mirror and orthogonal projection in the image plane as described above, the above relationships are determined as follows. In the system of coordinates {x,y,z}, for the stereographic ray projection which uses the ray passing through the north pole V of coordinates (0,0,1), point P of coordinates (x,y,z), and the point J with coordinates (u′,v′,0), one can write the join:
J(λ)=(1−λ)*V+λ*P
The above equation can be written explicitly as:
Therefore,
Using the results derived above, it follows that:
Next, in order to determine the coordinates (u′,v′) of projected points J corresponding to a sphere S of radius r, a scale factor is used to scale the coordinates (U′,v′) of the points J(u′,v′) computed above for the case of using a unit sphere (r=l). The parabolic mirror has a maximum vertical angle-of-view limit of α, above the horizontal plane that cuts the paraboloid M to create a finite mirror. As shown in
Therefore,
It follows that,
Assuming the omnidirectional image captured in E is circular, the same scaling factor applies for the v′ axis. Thus, the coordinates of points J(u′,v′) have to be scaled by a factor of r* A, to correspond to a sphere of radius r and an angle α.
Furthermore, the scaled points J (whose coordinates have been scaled by the factor u′s=v′s=r*A, are represented in a new coordinate system {u, v} with the origin at the upper left corner of the captured image E, as shown in
Finally, the common format image I has the structure shown in
The determination of each pixel in the image I can now be obtained by an interpolation of pixel values from the captured image E. The value of a pixel I(i; j) is determined by using the equation below in which the known relationship between coordinates x=x(i) and y =y(j) is utilized (see
Therefore, for a given pixel of coordinates (i; j) in image I, its value is determined by interpolation from pixel values in the image E, whose coordinates (u,v) are determined as specified in the previous equations. Thus,
I(i,j)=g[E(u,v), E(u,v+1), E(u+1,v), E(u+1,v+1)],
where g represents an interpolation operation.
Encoded omnidirectional images mapped into the representation format I obtained as described previously are decoded using a standard decoder, such as JPEG or MPEG. Once an image I has been decoded, it can be texture-mapped directly onto the faces of the octahedron X as shown in
The above described imaging process is summarized by the flow chart of
The mapping process includes the processes 304, 306, 308, 310 shown in
The packed image formed per step 310 is then encoded, transmitted, and decoded per steps 312 and 314 in the manner described previously. The decoded image is then displayed per step 316 in a well known manner.
An example of a perspective image generated when the above described mappings and encoding is performed with the image acquisition system of
The images I in the common representation format can be encoded and decoded using a standard video codec. The corresponding reconstructed images Î are rendered by the decoder and the visual information interactively displayed. An illustration of the process of texture mapping the reconstructed image Î to the octahedron X is shown in
Therefore, perspective views of the scene can be generated interactively by the decoder, from the image data transformed in the common representation format and corresponding to the panoramic or omni-directional-like visual information acquired by the imaging device. What this means is that a perspective view of the imaged scene can be generated and displayed, where the direction of view originating from the viewpoint (or center of projection O) can be changed interactively by the user (e.g., pan through the panoramic scene). This process can take place continuously during the decoding and display of perspective views from panoramic video data. A sample perspective view interactively generated from the image Î is shown in
The foregoing description is provided to illustrate the invention, and is not to be construed as a limitation. Numerous additions, substitutions and other changes can be made to the invention without departing from its scope as set forth in the appended claims. For example, variations on the image acquisition systems, representation and/or imaging techniques and the calibration techniques are possible without departing from the spirit of the invention.
In the case of alternative image acquisition systems, one of the main purposes of the present invention is to transform a panoramic or omni-directional-like image into a format that is not dependent on the type of image acquisition system used. Thus, the representation and imaging process described previously is equally applicable to a rotating camera that rotates about a single axis and produces a plurality of still images that are used to form the omnidirectional data. The present invention allows the captured image data from the still images to be projected onto the sphere S, so that all the other previously described and unaltered operations are performed.
In addition, variations of the representation processes are possible. For example, representation processing can be applied to video images where each frame of the video image has representations processed in the manner as described above.
Regarding variations of the calibration process, the examples given above generally describe the calibration and processing of a single image. Naturally, the calibration and image processing can be applied to video images where each frame of the video image is processed in the manner as described above.
Besides the calibration technique described previously, it is also contemplated that calibration of the frames in a video sequence be performed in a dynamic manner. Such a calibration process would involve capturing a plurality of frames of omni-directional-like video information via an image projection system similar to that shown in
1) predict parameters of the latter frame, 2) predict whether a calibration of the latter frame will fail or 3) increase computational precision for the computation of the required imaging device parameters.
Claims
1. A method for dynamically calibrating an image acquisition system from a plurality of frames of an omni-directional-like image, the method comprising:
- generating a first omni-directional-like image via an image acquisition system;
- generating first calibration information regarding said image acquisition system based on said first omni-directional-like image;
- generating information regarding a second omni-directional-like image generated by said image acquisition system, wherein said information is based on said first calibration information.
2. The method of claim 1, further comprising predicting parameters of said second omni-directional-like image based on said information.
3. The method of claim 1, further comprising increasing computational precision for the determination of a calibration parameter based on said information.
Type: Application
Filed: Jul 12, 2006
Publication Date: Nov 16, 2006
Inventors: Dan Lelescu (Morgan Hill, CA), Frank Bossen (San Jose, CA)
Application Number: 11/486,170
International Classification: G06K 9/36 (20060101);