SYSTEM AND METHOD FOR IMAGE REGISTRATION BASED ON VARIABLE REGION OF INTEREST
An image registration system for aligning first and second images. The novel system includes a first system for extracting a region of interest (ROI) from each image and a second system for coarsely aligning the regions of interest. The first system determines the size and location of the ROI based on the number of features contained within the region. The size of the ROI is enlarged until a number of features contained in the ROI is larger than a predetermined lower bound or until the size is greater than a predetermined upper bound. The second system computes a cross-correlation on the regions of interest using a plurality of transforms to find a coarse alignment transform having a highest correlation. The image registration system may also include a third system for performing sub-pixel alignment on the regions of interest.
Latest RAYTHEON COMPANY Patents:
1. Field of the Invention
The present invention relates to image processing systems. More specifically, the present invention relates to systems and methods for image registration.
2. Description of the Related Art
In applications such as remote sensing, medical imaging, and computer vision, multiple images of a scene or object are acquired at different times or from different perspectives. These images will typically be in different coordinate systems. Image registration is the process of transforming the images such that they are all in one coordinate system, allowing for the comparison or integration of data from the different images. The transformation used may include translation, rotation, scaling, or other more complicated transforms.
In general, image registration can be performed by trying different transforms on the target image (for example, translating the image using several different shifts), computing a correlation metric for each transform, and selecting and applying the transform having the highest correlation. This process, however, can become computationally expensive as the size of the image increases. The problem is even greater when sub-pixel registration is required (i.e., using image shifts of a fraction of a pixel).
A commonly used solution is the Fitts' algorithm, which uses a gradient-based approach to perform sub-pixel image registration. Gradient-based approaches are typically less computationally expensive than correlation approaches, but can be less accurate. The Fitts' algorithm is typically only accurate to within one pixel. As is known in the art, its performance is typically unsatisfactory for images having shifts of more than one pixel. Applying low pass filtering followed by downsampling prior to applying the Fitts' algorithm does not always yield satisfactory results. Low pass filtering and downsampling increases the dynamic range at the expense of precision. This low pass filtering and downsampling approach is limited by the amount of detail in the image. It is possible to remove valuable information with this approach and, from experimentation, it does not necessarily result in the correct image shift.
Hence, a need exists in the art for an improved system or method for sub-pixel image registration offering greater accuracy and shorter computation time than prior approaches.
SUMMARY OF THE INVENTIONThe need in the art is addressed by the image registration system for aligning a first image and a second image of the present invention. The novel image registration system includes a first system for extracting a region of interest (ROI) from each image and a second system for coarsely aligning the regions of interest. The first system determines the size and location of the ROI based on the number of features (such as edges) contained within the region. The size of the ROI is enlarged until a number of features contained in the ROI is larger than a predetermined lower bound or until the size is greater than a predetermined upper bound. The second system computes a cross-correlation on the regions of interest using a plurality of transforms to find a coarse alignment transform having a highest correlation. The image registration system may also include a third system for performing sub-pixel alignment on the regions of interest. The third system interpolates the regions of interest and performs a cross-correlation on the interpolated regions of interest using a plurality of transforms to find a sub-pixel alignment transform having a highest correlation. The coarse alignment and sub-pixel alignment transforms can then be applied to one (or both) of the original input images to align them.
Illustrative embodiments and exemplary applications will now be described with reference to the accompanying drawings to disclose the advantageous teachings of the present invention.
While the present invention is described herein with reference to illustrative embodiments for particular applications, it should be understood that the invention is not limited thereto. Those having ordinary skill in the art and access to the teachings provided herein will recognize additional modifications, applications, and embodiments within the scope thereof and additional fields in which the present invention would be of significant utility.
The image registration system 10 of the present invention reduces computation time by performing correlation processing on small sub-areas of the images, instead of on the entire images. The image registration system 10 includes a novel system 20 for extracting a “region of interest” (ROI) from each image. The location and size of the ROI are the same for both images, and are based on the amount of features contained in the ROI. In the illustrative embodiment, features are chosen to be edge (boundary) pixels, because it may be easier to correlate coarsely with edges than with smoother features. The teachings of the present invention, however, may be adapted to use other features such as curves, line intersections, high frequency components, etc. The size of the ROI is variable, depending on the number of features it contains. If an ROI does not contain enough features, it is enlarged. The growth of the ROI is stopped once it contains enough edge pixels to perform an accurate image registration. Other metrics may also be used to determine the size and location of the ROI.
After the ROI is found, a coarse alignment system 40 aligns the ROIs of the two images to within one pixel. The coarse alignment system 40 performs cross-correlation on the ROIs of the two images. Because the region of interest is smaller than the entire image, cross-correlation can be performed much faster than performing cross-correlation on the entire image. The image shift (including a shift sx in the horizontal direction and a shift sy in the vertical direction) corresponding to the highest correlation is noted. This shift is then used to coarsely align the two ROIs.
After the two ROIs are coarsely aligned, a fine alignment system 60 finds the sub-pixel shift between the images. The fine alignment system 60 interpolates the two ROIs and performs cross-correlation on the interpolated ROIs. The shift (px in the horizontal direction and py in the vertical direction) corresponding to the highest correlation is chosen as the sub-pixel shift of the images. The coarse and sub-pixel shifts can then be applied to the original images to align them.
As shown in
In the sample image of
Alternatively, for applications where the features are not always concentrated at the center of the image,
Next, select an ROI location based on the edge projections, where projections have the highest number of edge pixels. In the illustrative embodiment of
In the example of
The illustrative algorithm shown in
After the location of the ROI is found, the image registration system 10 determines the size of the ROI. There are several possible metrics that can be used to determine the ROI size. The ROI size should be small since a large ROI would result in more floating point operations (floating point multiplies, adds, etc.) and therefore greater computation time. Too small of an ROI, however, may not have enough features to perform an accurate image registration.
The ROI size is then grown until the number of features contained in the ROI is within an upper and lower bound. At Step 112, count the number of edge pixels contained within the current ROI using the edge map extracted from the first image in Step 22 of
If the number of edge pixels in the current ROI is greater than the lower bound, then the ROI size has been found. If the ROI does not contain enough edge pixels, then at Step 116, determine if the ROI size is larger than a predetermined maximum ROI size. This upper bound is included to prevent the ROI from growing too large, which may result in unacceptable computation times. If the number of edge pixels in the ROI is greater than the upper bound, then use the maximum ROI size. Alternatively, if the number of edge pixels in the ROI is greater than the upper bound, the algorithm can be adapted to search for a different ROI location (by, for example, adapting the ROI location algorithm 24 of
If at Step 116 the number of edge pixels in the ROI is less than the upper bound then, at Step 118, expand the ROI by, for example, increasing the ROI size by a predetermined number of pixels in each direction. Then repeat the process by returning to Step 112 (counting the number of edge pixels in the new ROI).
Returning to the system block diagram of
After the ROIs of the two input images are extracted, the coarse alignment system 40 performs cross-correlation on the ROIs to find a rough approximation of the image shift between the images. First, at Step 46, perform cross-correlation on the ROIs of the two images. The cross-correlation can be computed for all possible shifts (of one image relative to the other image) or, to same computation time, it can be limited to a predetermined number of shifts in each direction. Note that the correlation is performed on the ROIs of the images, not the entire images. This process uses fewer floating point operations compared to full-size image cross-correlation, and therefore reduces the overall computation time of the image registration process.
At Step 48, the shift (sx, sy) corresponding to the maximum correlation is selected as the coarse image shift, and at Step 50, the ROIs of the two images are coarsely aligned (i.e., by shifting one of the images relative to the second image by the coarse image shift).
After the ROIs are coarsely aligned, the fine alignment system 60 determines the sub-pixel shift between the images. At Step 62, interpolate the ROI of the first image by a predetermined integer N, and at Step 64, interpolate the ROI of the second image by N. If an original ROI size has dimensions R×C, then the interpolated ROI has dimensions NR×NC. The integer N determines the resolution of the sub-pixel image registration. The system is accurate to within 1/N pixels. A larger number N would increase the resolution of the system, but may also increase computation time. In the illustrative embodiment, N is set to three.
Next at Step 66, compute the cross-correlation of the interpolated ROIs. Because the two images are already coarsely aligned to within one (uninterpolated) pixel, the correlation does not need to be computed for all possible shifts. The sub-pixel shift can be determined by computing the correlation for only a few image shifts: N shifts to the right, N shifts to the left, N shifts up, and N shifts down.
At Step 68, the shift (px, py) corresponding to the maximum correlation is selected as the sub-pixel image shift. The final image shift (Δx, Δy) includes both the coarse and sub-pixel image shifts: (Δx, Δy)=(sx, sy)+(px, py)/N. At Step 70, apply the coarse and sub-pixel image shifts to the original input images to align the two images (i.e., by shifting one of the images relative to the second image by the final image shift). This step may include interpolating one of the images, shifting, and then downsampling to perform the sub-pixel shift.
The illustrative system described above assumes a translational (shift) difference between the two input images. The system can also be adapted for use with different transformation models. In general, coarse alignment can be performed by applying several different transforms (which may include one type of transform using different transform parameters, or it may include using several different types of transforms) to the ROIs of the input images, computing a correlation metric for each transform, and selecting the transform corresponding with the highest correlation to coarsely align the ROIs of the images. Similarly, fine alignment can be performed by interpolating the ROIs, applying several different transforms to the interpolated ROIs, computing a correlation metric for each transform, and selecting the transform corresponding with the highest correlation as the sub-pixel transform. The coarse alignment transform and sub-pixel transform can then be applied to the original images to align them.
Note that in accordance with the present teachings, correlations are performed on the ROIs of the images, not on the entire images. Image registration systems using cross-correlation typically perform better than gradient-based approaches but are more computationally expensive. By using ROIs, the number of floating-point operations can be reduced dramatically while maintaining accuracy.
The system can also be adapted to find multiple ROIs. The method shown in
Image registration using the novel method of the present invention may be much faster than conventional image registration approaches that correlate entire images, especially when the images contain few features. For improved performance, the systems and algorithms described above can be implemented in hardware using, for example, discrete logic circuits, FPGAs, ASICs, etc. Alternatively, the systems and algorithms can be implemented in software, stored in a memory device and executed by a processor. Other implementations may also be used without departing from the scope of the present teachings.
Thus, the present invention has been described herein with reference to a particular embodiment for a particular application. Those having ordinary skill in the art and access to the present teachings will recognize additional modifications, applications and embodiments within the scope thereof.
It is therefore intended by the appended claims to cover any and all such applications, modifications and embodiments within the scope of the present invention.
Accordingly,
Claims
1. A system for finding a region of interest in an image comprising:
- a first system for detecting features in said image;
- a second system for determining a location for said region of interest; and:
- a third system for determining a size of said region of interest based on a number of features contained within a region of said size and at said location in said image.
2. The invention of claim 1 wherein said third system includes a system for enlarging said size until a number of features contained in said region is larger than a predetermined lower bound or until said size is greater than a predetermined upper bound.
3. The invention of claim 1 wherein said location is fixed.
4. The invention of claim 3 wherein said location is at a center of said image.
5. The invention of claim 1 wherein said location is dependent on a number of features contained in said region of interest.
6. The invention of claim 5 wherein said second system includes a system for counting a number of features in each row and each column of said image.
7. The invention of claim 6 wherein said second system further includes a system for finding a first subset of rows having a large number of features.
8. The invention of claim 7 wherein said second system further includes a system for finding a second subset of columns having a large number of features.
9. The invention of claim 8 wherein said location is at an intersection of said first and second subsets.
10. A method for finding a region of interest in an image including the steps of: detecting features in said image;
- determining a location for said region of interest; and
- determining a size of said region of interest based on a number of features contained within a region of said size and at said location in said image.
11. The invention of claim 10 wherein said method includes enlarging said size until a number of features contained in said region is larger than a predetermined lower bound or until said size is greater than a predetermined upper bound.
12. The invention of claim 10 wherein said location is determined by counting a number of features in each row and each column of said image, finding a first subset of rows having a large number of features and a second subset of columns having a large number of features, and choosing said location at an intersection of said first and second subsets.
13. A method for aligning a first image and a second image including the steps detecting features in one of said images;
- determining a location for a region of interest;
- determining a size of said region of interest based on a number of features contained within a region of said size and at said location in said image;
- applying said size and location to said first and second images to extract a region of interest from each image; and
- cross-correlating said regions of interest using a plurality of transforms to find a first transform that coarsely aligns said regions of interest.
14. The invention of claim 13 wherein said method further includes interpolating said regions of interest.
15. The invention of claim 14 wherein said method further includes crosscorrelating the interpolated regions of interest using a plurality of transforms to find a second transform that sub-pixel aligns said regions of interest.
16. The invention of claim 15 wherein said method further includes applying said first and second transforms to one or both of said first and second images.
Type: Application
Filed: Mar 14, 2012
Publication Date: Jul 5, 2012
Applicant: RAYTHEON COMPANY (Waltham, MA)
Inventors: Eric P. Lam (Diamond Bar, CA), Christopher A. Leddy (Huntington Beach, CA), Harrison A. Parks (Los Angeles, CA), Stephen R. Nash (Playa Del Rey, CA)
Application Number: 13/419,668
International Classification: G06K 9/46 (20060101);