Hybrid recursive analysis of spatio-temporal objects
A method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives has: providing at least two hypotheses for the 3D-information; performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion; updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion; and determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis. A corresponding computer readable digital storage medium and a 3D-information generator are also described. Further embodiments perform a correspondence analysis between projections of spatio-temporal objects (STO) in multiple images to select a particular spatio-temporal object on the basis of said correspondence analysis.
Latest FRAUNHOFER-GESELLSCHAFT ZUR FOERDERUNG DER ANGEWANDTEN FORSCHUNG e.V. Patents:
- CLOSED SOUND RECEIVER WITH SOUND-PERMEABLE BOUNDARY SURFACE
- Method for controlling a driver circuit, driver circuit, system comprising a driver circuit and method for manufacturing an integrated circuit
- Noise filling in multichannel audio coding
- Concept for picture/video data streams allowing efficient reducibility or efficient random access
- Vehicle for transporting cargo
This application is a continuation of copending International Application No. PCT/EP2012/073506, filed Nov. 23, 2012, which is incorporated herein by reference in its entirety, and additionally claims priority from U.S. Provisional Application No. 61/564,916, filed Nov. 30, 2011, which is also incorporated herein by reference in its entirety.
BACKGROUND OF THE INVENTIONOne major objective of computer vision algorithms is to create spatial and temporal consistent 3D information from multiple cameras and/or multi-sensor setups.
Patch sweeping is an extension of the well-known plane sweeping approaches. In recent years, plane sweeping was considered as one of the most promising approaches for real-time depth estimation, but the estimation quality is limited (Colins, R. T. (1996): “A Space-Sweep approach to true Multi-Image matching”, Computer Vision and Pattern Recognition, IEEE Computer Society Conference, vol. 0 p. 358).
Simultaneously, patch based techniques (Furukawa, Y. Ponce, J. (2007): “Accurate, dense, and robust Multi-View stereopsis”, Computer Vision and Pattern Recognition, CVPR 07) turned out to be among the most accurate but time consuming 3D reconstruction methods (Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R. (2006): “A comparison and evaluation of Multi-View Stereo reconstruction algorithms”, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society). Inspired by the contrary properties of the plane sweeping and patch based approaches, patch sweeping emerged as a symbiotic fusion of these two techniques.
Patch sweeping assumes that a 3D object surface can be described with quadratic surface elements, which are also known as surflets or spatial patches. In order to estimate the depth of an object for a reference camera or the object surface, several oriented spatial patches along the viewing rays of this reference camera have to be tested (see
An extended hybrid recursive matching (HRM) was proposed by N. Atzpadin, P. Kauff, O. Schreer in 2004 in the article “Stereo Analysis by Hybrid Recursive Matching for Real-Time Immersive Video Conferencing”, IEEE Trans. on Circuits and Systems for Video Technology, Special Issue on Immersive Telecommunications, Vol. 14, No. 4, 321-334. European Patent Application EP 1 229 741 A1 by Atzpadin, N., Karl, M., Kauff, P., Schreer, O. (2002) describes a “Process for Real-Time Correspondence Analysis of Image Features in Corresponding Video Images”. The article “True-Motion Estimation with 3-D Recursive Search Block Matching” by De Haan, G., Biezen, P. W. A. C., Huijgen, H., Ojo, O. A., IEEE Trans. on Circuits and Systems for Video Technology, (1993), Vol. 3, No. 5, 368-379 describes further related research.
One major challenge of conventional patch sweeping is to define the search range in the parameter space in order to get results of high depth resolution but with a reasonable computational effort.
SUMMARYAccording to an embodiment, a method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives may have the steps of: providing at least two hypotheses for the 3D-information; performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion, wherein the multi-hypotheses test has determining projections of at least two 3D spatio-temporal objects defined by the at least two hypotheses on a plurality of image planes corresponding to the multiple images and performing a correspondence analysis between the projections in the multiple images to determine the test-result hypothesis; updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion; and determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis.
According to another embodiment, a 3D-information generator may have: an interface for receiving multiple images showing a 3D scene from multiple perspectives; a hypotheses provider for providing at least two hypotheses for the 3D-information; a multi-hypotheses tester for performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion, the multi-hypotheses tester having a projection calculator for determining projections of at least two 3D spatio-temporal objects defined by the at least two hypotheses on a plurality of image planes corresponding to the multiple images, and a correspondence analyzer for performing a correspondence analysis between the projections in the multiple images to determine the test result hypothesis; a hypothesis updater for updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve a fulfillment of the matching criterion or another criterion; and a 3D-information interface for providing the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the hypothesis updater.
According to still another embodiment, a method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives may have the steps of: providing a plurality of spatio-temporal object parameter sets, each spatio-temporal object parameter set having object parameters describing a 3D spatio-temporal object of a plurality of 3D spatio-temporal objects; determining projections of the plurality of spatio-temporal objects on a plurality of image planes corresponding to the multiple images using the plurality of spatio-temporal object parameter sets; performing a correspondence analysis between the projections in the multiple images; selecting a particular spatio-temporal object parameter set from the plurality of spatio-temporal object parameter sets on the basis of a result of the correspondence analysis; and determining the 3D-information on the basis of the object parameters of the particular spatio-temporal object parameter set.
According to another embodiment, a 3D-information generator for generating 3D-information from multiple images showing a 3D scene from multiple perspectives may have: a parameter set provider for providing a plurality of spatio-temporal object parameter sets, each spatio-temporal object parameter set having object parameters describing a 3D spatio-temporal object of a plurality of 3D spatio-temporal objects; a projection calculator for determining projections of the plurality of spatio-temporal objects on a plurality of image planes corresponding to multiple images using the plurality of spatio-temporal object parameter sets; a correspondence analyzer for performing a correspondence analysis between the projections in the multiple images; a parameter set selector for selecting a particular spatio-temporal object parameter set from the plurality of spatio-temporal object parameter sets on the basis of a result of the correspondence analysis; and a 3D-information interface for providing the 3D-information on the basis of the object parameters of the particular spatio-temporal object parameter set.
Another embodiment may have a computer readable digital storage medium having stored thereon a computer program having a program code for performing, when running on a computer, the above methods for generating 3D-information from multiple images.
Embodiments of the present invention provide a method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives. The method comprises providing at least two hypotheses for the 3D-information and performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion. The method further comprises updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion. The method also comprises determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis.
Further embodiments provide a computer readable digital storage medium having stored there on a computer program having a program code for performing, when running on a computer, a method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives. The method comprises providing at least two hypotheses for the 3D-information and performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion. The method further comprises updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion. The method also comprises determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis.
Further embodiments of the present invention provide a 3D-information generator. The 3D-information generator comprises an interface for receiving multiple images showing a 3D scene from multiple perspectives, a hypotheses provider for providing at least two hypotheses for the 3D-information, and a multi-hypotheses tester for performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion. The 3D-information generator further comprises a hypothesis updater for updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion. The 3D-information generator also comprises a 3D-information interface for providing the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the hypothesis updater.
According to further embodiments, a method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives comprises: providing a plurality of spatio-temporal object parameter sets, each spatio-temporal object parameter set comprising object parameters describing a 3D spatio-temporal object of a plurality of 3D spatio-temporal objects. The method further comprises determining projections of the plurality of spatio-temporal objects on a plurality of image planes corresponding to the multiple images using the plurality of spatio-temporal object parameter sets. The method also comprises performing a correspondence analysis between the projections in the multiple images. Furthermore, the method comprises selecting a particular spatio-temporal object parameter set from the plurality of spatio-temporal object parameter sets on the basis of a result of the correspondence analysis. The 3D-information is then determined on the basis of the object parameters of the particular spatio-temporal object parameter set.
Further embodiments relate to a computer-readable digital storage medium that has stored thereon a computer program having a program code for performing, when running on a computer, the method mentioned above for generating 3D-information from multiple images showing a 3D-scene from multiple perspectives.
According to further embodiments, a 3D-information generator is provided which comprises a parameter set provider for providing a plurality of spatio-temporal object parameter sets. Each spatio-temporal object parameter set comprises object parameters describing a 3D spatio-temporal object of a plurality of 3D spatio-temporal objects. The 3D-information generator further comprises a projection calculator for determining projections of the plurality of spatio-temporal objects on a plurality of images planes corresponding to the multiple images using the plurality of spatio-temporal object parameter sets. A correspondence analyzer is provided for performing a correspondence analysis between the projections in the multiple images. The 3D-information generator also comprises a parameter set selector for selecting a particular spatio-temporal object parameter set from the plurality of spatio-temporal object parameter sets on the basis of a result of the correspondence analysis. Furthermore, the 3D-information generator comprises a 3D-information interface for providing the 3D-information on the basis of the object parameters of the particular spatio-temporal object parameter set.
The presented method exploits the advantages of a fast hybrid recursive disparity estimation scheme with the accuracy offered by a patch sweeping approach. Moreover, conventional blocks from block-matching are replaced by spatio-temporal objects, which offer a more general representation of the 3D scene.
As mentioned above, one major challenge of conventional patch sweeping is to define the search range in the parameter space in order to get results of high depth resolution but with a reasonable computational effort. This search range problem is solved by the hybrid recursive analysis, which is instead based on a set of a few hypotheses lowering the computational effort significantly.
Definition of a Spatio-Temporal Object
For the purpose of the subsequent description, a so-called spatio-temporal object (STO) is introduced, which is considered to be an extension of standard patches mentioned above. A spatio-temporal object is a time varying description of an arbitrary 3D shape containing parameters such as 3D position, 3D orientation, deformation and material properties like object transparency or reflectance and possibly many more. Due to the time-varying nature of an STO, it can be also considered as an arbitrary formed and deformable time-dependent 3D object. Hence, a STO represents a temporarily varying 3D Object defined by a parameter vector. In order to compare a STO with 2D projections in different images, the projection of a STO is used.
Hybrid Recursive Analysis Based of Spatio-Temporal Objects
The hybrid recursive analysis is an extension of the hybrid recursive matching (HRM), which was developed at Fraunhofer HHI (Heinrich Hertz Institut). The HRM is defined for a stereo setup dealing with disparity vectors. In contrast to this, the proposed hybrid recursive analysis enhances stereo to multi-view setups and uses a more general definition of so-called spatio-temporal objects (STO) instead of disparities.
Hence, the proposed algorithm replaces the test of several patches with a large number of parameter combinations like position, orientation, deformation etc. by an efficient multi hypotheses test of only a limited, relatively small number of hypotheses (or corresponding spatio-temporal object parameter sets) in order to exploit the fact that the patches usually do not change significantly in space and time. The 3D patches in the local neighborhood and from the former image define these new hypotheses described by a parameter vector.
The hybrid recursive analysis uses previous processing results in order to create a spatial and temporal consistent representation of the 3D scene. This concept of recursive analysis avoids high search ranges via an efficient hypothesis analysis. The rationale is that such hypotheses are the most likely to provide a good estimate of the 3D structure for the current pixel and its neighbourhood. Hypotheses are defined by STOs, which offers per definition temporal and spatial properties.
An example of the overall hybrid recursive analysis based on hypotheses of spatio-temporal objects is depicted in
The validation of a limited number of given hypothesis is evaluated in the multi hypotheses test 24. Typically at least two hypotheses for the 3D-information are provided to the multi-hypotheses test 24 in order to be tested against the information contained in the multiple views/images. In the 3D-information generator, a hypotheses provider 22 (in the case of
After that, a hypotheses update procedure 27 delivers a new hypothesis, resulting from a multi-dimensional optimization step, as well as based on a hypothesis confidence measure. For example, the hypotheses update procedure may vary the parameter set of the test-result hypothesis to further improve the matching criterion (i.e., to obtain a higher/better degree of matching between the updated test-result hypothesis and the actual 3D object in the 3D scene acquired by the camera). The 3D-information generator comprises a hypothesis updater for updating the test-result hypothesis. This additional hypothesis is tested against the best hypothesis from multi hypotheses test 24. This is typically unavoidable due to the fact that the algorithm has to initialize and the hypotheses have to be adapted to the current image especially in regions with complex 3D structure or in high dynamic regions. In particular, an action of comparing an updated test-result and the test-result hypothesis with respect to the particular matching criterion or a further matching criterion (e.g., the confidence) may be performed. The updated test-result hypothesis or the test-result hypothesis may then be chosen as the resulting hypothesis in dependence on a result of the comparison.
The 3D-information may then be determined on the basis of the parameter set of a resulting hypothesis (final hypothesis in
The whole algorithm can be divided into three stages:
1. In the multi hypotheses test 24, a small number of hypotheses are evaluated for the current position;
2. the hypothesis with the best result is chosen as the start for the hypothesis update and refinement procedure 27, which yields an additional hypothesis;
3. the final result is obtained by comparing the hypothesis from the update procedure with the best hypothesis yielding from the multi hypotheses test.
Only a very limited number of hypotheses are tested in the hypotheses test 24 to find the best hypothesis of a STO for the actual position. In contrast to standard algorithms no search area is defined. In contrast to the stereo HRM the hypotheses are no longer checked in a single image pair but tested in all existing camera images in parallel.
The hybrid analysis scheme has two main advantages in comparison to common approaches. The recursive structure speeds up the analysis dramatically. The combined choice of spatial and temporal hypotheses yields spatially and temporally consistent 3D structure due to an efficient strategy of testing particular hypotheses.
As can be seen in
It may be possible, in some embodiments, that the hypothesis updater 37 provides the resulting hypothesis to the hypothesis provider 32 so that the resulting hypotheses can be provided to the multi-hypo thesis tester 34 during a subsequent recursion performed by the 3D-information generator 30.
According to a further aspect of the teachings disclosed herein, a method for generating 3D information from multiple images showing a 3D scene from multiple perspectives comprises providing a plurality of spatio-temporal object parameter sets, each spatio-temporal parameter set comprising object parameters describing the 3D spatio-temporal object of a plurality of 3D spatio-temporal objects. This provision of the spatio-temporal object parameter sets may be performed by a parameter set provider 42 of a 3D-information generator. 40, as schematically illustrated in block diagram form in
The method further comprises a determination of projections (for example, by means of a projection calculator 43) of the plurality of spatio-temporal objects on a plurality of image planes corresponding to the multiple images using the plurality of spatio-temporal object parameter sets. The image planes may be described by image plane descriptions provided by data record 46 describing the acquisition geometry. Then a performance analysis between the projections in the multiple images is performed. A particular spatio-temporal object parameter set is selected from the plurality of spatio-temporal object parameter sets on the basis of the a result of the correspondence analysis. The actions of determining the projections, of performing the correspondence analysis (for example, using a correspondence analyzer 44, as shown in
Each of 3D spatio-temporal objects may be regarded as a hypothesis. The plurality of spatio-temporal parameter sets therefore corresponds to a plurality of hypotheses for a spatially and temporally limited segment of the 3D scene. The plurality of hypotheses may then be analyzed by means of the multi-hypotheses test.
Aspects of at Least Some Embodiments3D Spatio-Temporal Objects (STO)
-
- A STO is an arbitrary shaped object in the 3D space
- A STO may consist on a 3D position, 3D orientation, material properties such as transparency and surface reflectance
- A STO is time dependent and therefore a deformable 3D object
Hypothesis Definition - A hypothesis is represented by a STO
- STOs are defined by a given parameter vector
Replacement of Block Matching by a Patch Sweep Based Approach - Each hypothesis represented by a STO is projected to the image planes of two or multiple cameras
- The STO projections in the images planes may be simplified versions
- The probability of a given hypothesis is determined by correspondence analysis of the projections of the STO in the image planes for two or multiple images simultaneously
Hypothesis Selection and Refinement - The best hypothesis is selected based on hypothesis confidence
- The hypothesis update is performed by a multi-dimensional parameter optimization of the parameter vector of a given hypothesis
- The optimization criteria is the related hypothesis confidence, describing the existence probability of a hypothesis
Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.
Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.
Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.
Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary.
A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.
A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.
In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods may be performed by any hardware apparatus.
While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which will be apparent to others skilled in the art and which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.
Claims
1. A method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives, the method comprising:
- providing at least two hypotheses for the 3D-information;
- performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion by selecting one of the at least two hypotheses as the test result hypothesis, wherein the multi-hypotheses test comprises determining projections of at least two 3D spatio-temporal objects defined by the at least two hypotheses on a plurality of image planes corresponding to the multiple images and performing a correspondence analysis between the projections in the multiple images to determine the test-result hypothesis;
- updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion, the parameter set comprising at least one of a 3D position, a 3D orientation, a deformation, and material properties;
- determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis; and
- repeating the steps of providing, performing and updating as a subsequent iteration,
- wherein the step of providing in the subsequent iteration comprises deriving at least one of the at least two hypotheses for the 3D-information from the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration, wherein the deriving comprises at least one of spatially translating, re-orienting and deforming the 3D spatio-temporal object of the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration.
2. The method according to claim 1, wherein the action of updating the test-result hypothesis further comprises:
- comparing an updated test-result hypothesis and the test-result hypothesis with respect to the particular matching criterion or a further matching criterion; and
- choosing the updated test-result hypothesis or the test-result hypothesis as the resulting hypothesis in dependence on a result of the comparison.
3. The method according to claim 1, wherein the provision of the at least two hypotheses comprises retrieving selected hypotheses relative to one or more previous recursion(s) of the method from a hypotheses memory, the one or more previous recursion(s) comprising a defined or known spatial and temporal relation to a current recursion of the method.
4. The method according to claim 3, wherein the retrieved selected hypotheses are a result of a recursion of the one or more previous recursion(s) thereby establishing a feed-back loop that includes the step of performing and the step of updating.
5. The method according to claim 1, wherein the update of the test-result hypothesis comprises a multi-dimensional parameter optimization of the parameter set of the test-result hypothesis.
6. The method according to claim 1, wherein the update of the test-result hypothesis comprises a recursive determination of parameter variations on the basis of at least one or more of:
- a test-result hypothesis determined during a previous recursion of the method and an updated test-result hypothesis determined during the previous recursion.
7. The method according to claim 1, wherein the particular matching criterion is at least one of a hypothesis confidence and a probability of a given hypothesis.
8. The method according to claim 1, wherein the projections of the at least two 3D objects are simplified versions.
9. The method according to claim 1, wherein the at least two hypotheses define at least one of the 3D position, the 3D orientation, a material property, and a surface reflectance of at least two corresponding 3D spatio-temporal objects.
10. The method according to claim 1,
- wherein the at least two hypotheses define at least two 3D spatio-temporal objects; and
- wherein the update of the test-result hypothesis comprises updating the parameter set according to the deformation of a corresponding 3D spatio-temporal object between a previous recursion and a current recursion of the method, the previous recursion regarding a temporally previous version of the corresponding 3D spatio-temporal object.
11. The method according to claim 1, wherein within the multi-hypotheses test a relatively small number of hypotheses is evaluated.
12. The method according to claim 1, wherein at least one of the at least two hypotheses is a result of a recursion, the recursion including the step of performing and the step of updating, thereby establishing a feed-back loop.
13. The method according to claim 1, wherein the deriving is performed so that at least two of the at least two hypotheses for the 3D-information are derived from the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration.
14. The method according to claim 1, wherein the step of providing in the subsequent iteration includes deriving at least a further one of the at least two hypotheses for the 3D-information from 3D information concerning a previous video frame.
15. A non-transitory computer readable digital storage medium having stored thereon a computer program comprising a program code for performing, when running on a computer, a method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives, the method comprising:
- providing at least two hypotheses for the 3D-information;
- performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion by selecting one of the at least two hypotheses as the test result hypothesis, wherein the multi-hypotheses test comprises determining projections of at least two 3D spatio-temporal objects defined by the at least two hypotheses on a plurality of image planes corresponding to the multiple images and performing a correspondence analysis between the projections in the multiple images to determine the test-result hypothesis;
- updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion, the parameter set comprising at least one of a 3D position, a 3D orientation, a deformation, and material properties;
- determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis; and
- repeating the steps of providing, performing and updating as a subsequent iteration,
- wherein the step of providing in the subsequent iteration comprises deriving at least one of the at least two hypotheses for the 3D-information from the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration, wherein the deriving comprises at least one of spatially translating, re-orienting and deforming the 3D spatio-temporal object of the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration.
16. A 3D-information generator comprising:
- an interface that receives multiple images showing a 3D scene from multiple perspectives;
- a hypotheses provider that provides at least two hypotheses for the 3D-information;
- a multi-hypotheses tester that performs a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion by selecting one of the at least two hypotheses as the test result hypothesis, the multi-hypotheses tester comprising a projection calculator for determining projections of at least two 3D spatio-temporal objects defined by the at least two hypotheses on a plurality of image planes corresponding to the multiple images, and a correspondence analyzer that performs a correspondence analysis between the projections in the multiple images to determine the test result hypothesis;
- a hypothesis updater that updates the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve a fulfillment of the matching criterion or another criterion, the parameter set comprising at least one of a 3D position, a 3D orientation, a deformation, and material properties; and
- a 3D-information interface that provides the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the hypothesis updater; wherein
- the 3D-information generator is configured so that, as a subsequent iteration, the hypotheses provider repeats the provision, the multi-hypotheses tester repeats the provision, and the hypothesis updater repeats the update;
- the hypothesis provider is configured to, in the subsequent iteration, derive at least one of the at least two hypotheses for the 3D-information from the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration, wherein the deriving comprises at least one of spatially translating, re-orienting and deforming the 3D spatio-temporal object of the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration; and
- each of the hypotheses provider, multi-hypotheses tester, and hypothesis updater is implemented on a computer, a microprocessor, or an electronic circuit.
17. The 3D-information generator according to claim 16, wherein the hypothesis updater comprises a hypotheses comparator that compares the updated test-result hypothesis and the test-result hypothesis, wherein the resulting hypothesis is chosen among the updated test-result hypothesis and the test-result hypothesis in dependence on an output of the comparator.
18. The 3D-information generator according to claim 16, further comprising a hypotheses memory that stores hypotheses relative to one or more previous recursion(s) of the method, wherein the hypotheses provider is further configured that retrieves selected hypotheses from the hypotheses memory, the selected hypotheses being related to one or more previous recursion(s) comprising a defined or known spatial and temporal relation to a current recursion of the method.
19. The 3D-information generator according to claim 16, wherein the hypothesis updater comprises a multi-dimension parameter optimizer that optimizes the parameter set of the test-result hypothesis.
20. The 3D-information generator according to claim 16,
- wherein the at least two hypotheses define at least two 3D spatio-temporal objects; and
- wherein the hypothesis updater updates the parameter set according to the deformation of a corresponding 3D spatio-temporal object between a previous recursion and a current recursion, the previous recursion regarding a temporally previous version of the corresponding 3D spatio-temporal object.
20060050952 | March 9, 2006 | Blais |
- Feldmann et al, “Multi-view depth estimation based on visual-hull enhanced Hybrid Recursive Matching for 3D video conference systems”, IEEE, 2009, pp. 745-748.
- Zimmermann et al, “Multiview 3D Tracking with an Incrementally Constructed 3D Model”, IEEE, 2006, pp. 1-8.
- Official Communication issued in International Patent Application No. PCT/EP2012/073506, dated Mar. 15, 2013.
- Feldmann et al., “Multi-View Depth Estimation Based on Visual-Hull Enhanced Hybrid Recursive Matching for 3D Video Conferences Systems”, 2009 16th IEEE International Conference on Image Processing (ICIP), IEEE, Nov. 7, 2009, pp. 745-748.
- Waizenegger et al., “Patch-Sweeping with Robust Prior for High Precision Depth Estimation in Real-Time Systems”, 2009 16th IEEE International Conference on Image Processing (ICIP), IEEE, Sep. 11, 2011, pp. 881-884.
- Waizenegger et al., “Real-time Patch Sweeping for High-Quality Depth Estimation in 3D Videoconferencing Applications”, Proceedings of SPIE, vol. 7871, Feb. 10, 2011, p. 7871E-1-7871E-10.
- Atzpadin et al., “Stereo Analysis by Hybrid Recursive Matching for Real-Time Immersive Video Conferencing”, IEEE Trans. on Circuits and Systems for Video Technology, Special Issue on Immersive Telecommunications, vol. 14, No. 4, Mar. 2004, pp. 321-334.
Type: Grant
Filed: May 27, 2014
Date of Patent: Apr 17, 2018
Patent Publication Number: 20140270486
Assignee: FRAUNHOFER-GESELLSCHAFT ZUR FOERDERUNG DER ANGEWANDTEN FORSCHUNG e.V. (Munich)
Inventors: Nicole Atzpadin (Neuendorf), Wolfgang Waizenegger (Berlin), Oliver Schreer (Berlin), Ingo Feldmann (Berlin), Peter Kauff (Berlin), Peter Eisert (Berlin)
Primary Examiner: Edward Park
Application Number: 14/287,274
International Classification: G06T 1/00 (20060101); G06T 7/00 (20170101); G06T 7/70 (20170101); G06T 7/593 (20170101);