IMAGE PROCESSING APPARATUS, IMAGE CAPTURING APPARATUS, IMAGE PROCESSING METHOD, AND STORAGE MEDIUM
There is provided an image processing apparatus. A determination unit determines a movement direction of an object. A setting unit sets, within a shooting range, a plurality of processing areas that are arranged in a different direction from the movement direction. A selection unit selects a tracking point in each processing area of a predetermined shot image. A tracking unit tracks, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image.
The present invention relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a storage medium.
Description of the Related ArtIn order to perform image stabilization on video shot using an image capturing apparatus such as a digital video camera, it is necessary to detect the amount of motion between frame images and perform position alignment on a plurality of images. As methods of detecting the amount of motion between frame images, there are methods that use information of external devices such as gyro sensors and methods that estimate the amount of motion from shot frame images.
Methods of estimating the amount of motion using frame images have been variously proposed heretofore, with motion vector detection through template matching being one such typical method. In template matching, first one of two frame images included in video is taken as an original image and the other frame image is taken as a reference image. A rectangular area of predetermined size arranged on the original image is then taken as a template block, and the correlation with the distribution of pixel values within the template block is derived at various positions of the reference image. At this time, the position at which the correlation is highest in the reference image is the movement destination of the template block, and the direction and movement amount to the movement destination when based on the position of the template block on the original image will be the motion vector.
In order to improve the detection rate of motion vectors, there is a technology that involves extracting a feature point from a frame image, and performing template matching by arranging the template block on the extracted feature point. Here, when feature point extraction is performed on an entire image, the distribution of the feature points will often be non-uniform. In the case where motion vectors obtained with respect to non-uniform feature points are used in the application of image stabilization, image stabilization will mainly be performed on the area in which the feature is concentrated. In view of this, as disclosed in Japanese Patent Laid-Open No. 2008-192060, in order to distribute feature points uniformly, a technology is known that involves dividing an image into grid squares, calculating a feature value representing the size of the feature for every pixel, and extracting the pixel having the largest feature value within each grid square as a feature point.
Also, as disclosed in Japanese Patent Laid-Open No. 2007-334625, in order to improve the detection rate of motion vectors, a technology for tracking feature points is known. Feature point tracking can be realized by sequentially detecting the motion vector of a feature point extracted from an image over a plurality of consecutive frame images.
In the processing of the next frame, the image capturing apparatus implements template matching centered at a tracking destination feature point 1402 that is obtained by adding the calculated vector value 1407 to the feature point 1401. Thereafter, the image capturing apparatus repeats, on the tracking destination feature point, the processing for adding the vector value and detecting the next tracking destination feature point, over a plurality of frames.
Also, as disclosed in Japanese Patent No. 4504158 and Japanese Patent Laid-Open No. 2012-73997, technologies for restricting the range of tracking based on the movement direction of the tracking target are also known.
However, the above-described patent documents do not describe the case in which the feature point tracking processing is performed on a plurality of processing areas.
SUMMARY OF THE INVENTIONThe present invention has been made in view of such circumstances, and provides a technology that, in the case of tracking feature points inside each of a plurality of processing areas in a shooting range, reduces the possibility of feature points moving in the direction of another processing area and tracking failure (loss of feature points) occurring.
According to a first aspect of the present invention, there is provided an image processing apparatus comprising: a determination unit configured to determine a movement direction of an object; a setting unit configured to set, within a shooting range, a plurality of processing areas that are arranged in a different direction from the movement direction; a selection unit configured to select a tracking point in each processing area of a predetermined shot image; and a tracking unit configured to track, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image.
According to a second aspect of the present invention, there is provided an image processing apparatus comprising: a setting unit configured to set a plurality of processing areas within a shooting range, and to set a plurality of selection areas within each processing area; a selection unit configured to select a tracking point in each selection area of each processing area of a predetermined shot image; and a tracking unit configured to track, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image, wherein, in a case where, in a specific shot image of the one or more shot images, the tracking unit fails in tracking of a specific tracking point selected in a specific processing area of the predetermined shot image, the selection unit selects a new tracking point to be substituted for the specific tracking point in a specific selection area among the plurality of selection areas of the specific processing area of the specific shot image, and the specific selection area is one of one or more selection areas that are next to or at least partially overlap with another processing area.
According to a third aspect of the present invention, there is provided an image capturing apparatus comprising: the image processing apparatus according to the first aspect; and an image capturing unit configured to generate the predetermined shot image and the one or more shot images.
According to a fourth aspect of the present invention, there is provided an image capturing apparatus comprising: the image processing apparatus according to the second aspect; and an image capturing unit configured to generate the predetermined shot image and the one or more shot images.
According to a fifth aspect of the present invention, there is provided an image processing method comprising: determining a movement direction of an object; setting, within a shooting range, a plurality of processing areas that are arranged in a different direction from the movement direction; selecting a tracking point in each processing area of a predetermined shot image; and tracking, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image.
According to a sixth aspect of the present invention, there is provided an image processing method comprising: setting a plurality of processing areas within a shooting range, and to set a plurality of selection areas within each processing area; selecting a tracking point in each selection area of each processing area of a predetermined shot image; and tracking, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image, wherein, in a case where, in a specific shot image of the one or more shot images, tracking of a specific tracking point selected in a specific processing area of the predetermined shot image fails, selecting a new tracking point to be substituted for the specific tracking point in a specific selection area among the plurality of selection areas of the specific processing area of the specific shot image, and the specific selection area is one of one or more selection areas that are next to or at least partially overlap with another processing area.
According to a seventh aspect of the present invention, there is provided a non-transitory computer-readable storage medium which stores a program for causing a computer to execute an image processing method comprising: determining a movement direction of an object; setting, within a shooting range, a plurality of processing areas that are arranged in a different direction from the movement direction; selecting a tracking point in each processing area of a predetermined shot image; and tracking, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image.
According to an eighth aspect of the present invention, there is provided a non-transitory computer-readable storage medium which stores a program for causing a computer to execute an image processing method comprising: setting a plurality of processing areas within a shooting range, and to set a plurality of selection areas within each processing area; selecting a tracking point in each selection area of each processing area of a predetermined shot image; and tracking, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image, wherein, in a case where, in a specific shot image of the one or more shot images, tracking of a specific tracking point selected in a specific processing area of the predetermined shot image fails, selecting a new tracking point to be substituted for the specific tracking point in a specific selection area among the plurality of selection areas of the specific processing area of the specific shot image, and the specific selection area is one of one or more selection areas that are next to or at least partially overlap with another processing area.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Hereinafter, embodiments of the present invention will be described with reference to the attached drawings. Elements that are given the same reference numerals throughout all of the attached drawings represent the same or similar elements. Note that the technical scope of the present invention is defined by the claims, and is not limited by the following respective embodiments. Also, not all of the combinations of the aspects that are described in the embodiments are necessarily essential to the present invention. Also, the aspects that are described in the individual embodiments can be combined as appropriate.
The following embodiments relate to cases where feature point tracking processing is performed independently on each of a plurality of areas that are included in the shooting range. The case where the image capturing apparatus is provided with a plurality of motion vector detection units that are constituted by hardware for performing feature point tracking processing is given as an example of such a case.
For example, as shown in
The loss of feature points will be further described in detail, with reference to
In this way, in the case where a plurality of processing areas are arranged in the same direction as the movement direction of an object, the possibility of feature points moving in the direction of another processing area and tracking failure (loss of feature points) occurring increases. As a result, a problem arises in that motion vector detection performance decreases.
First EmbodimentA CPU 112 is constituted by a microcomputer or the like that administers overall control of the digital camera 100, issues operation instructions to the various functional blocks, and executes various types of control processing. A bus 114 is a system bus, and a bus 115 is an image data bus.
A DRAM 107 is a memory for storing data, and is provided with sufficient storage capacity to store the data of a predetermined number of still images, a predetermined length of video, audio and the like, constants for use in operations by the CPU 112, computer programs, and the like. A memory control unit 106 performs data writing to and data readout from the DRAM 107, according to instructions from the CPU 112 or a data transfer unit 105.
A nonvolatile memory control unit 108 performs data writing to and data readout from a ROM 109, according to instructions from the CPU 112. The ROM 109 is an electrically erasable recordable memory, and an EEPROM or the like is used. Constants for use in operations by the CPU 112, computer programs and the like are stored in the ROM 109.
The CPU 112 controls an image processing unit 104, the data transfer unit 105, the memory control unit 106, the nonvolatile memory control unit 108, a display control unit 110, an operation unit 113 and the image sensor 102 via the bus 114. The CPU 112 realizes the various processing of the present embodiment, by executing computer programs recorded in the ROM 109.
A display unit 111 consists of a liquid crystal display monitor or the like, is controlled by the display control unit 110, and displays various types of image data and the like. The operation unit 113 includes switches, buttons and the like that are operated by a user, and is used in operations such as power ON/OFF, shutter ON/OFF, and the like.
The image processing unit 104 is constituted by various types of image processing circuits, a buffer memory and the like, and is provided with a motion vector detection unit 121, a motion vector detection unit 122, and an object detection unit 123. The motion vector detection unit 121 and the motion vector detection unit 122 have the same functions, and perform motion vector detection independently. The object detection unit 123 detects motion amounts and feature amounts of a face portion or a body portion of objects, and detects a main object.
The data transfer unit 105 is constituted by a plurality of DMACs (Direct Memory Access Controllers) that perform data transfer. A gyro sensor 116 detects the attitude (horizontally positioned, vertically positioned, etc.) and movement direction of the digital camera 100, and notifies the detection results to the CPU 112.
A set number of feature point extraction grid squares 302 (white grid squares in
Next, feature point tracking processing will be described, with reference to
In step S400, the CPU 112 determines the direction for dividing the processing area based on information from the gyro sensor 116 and the object detection unit 123, and sets the determined division direction in a division direction setting unit 260. The division direction setting unit 260 sets a target processing area within the shooting range, based on the set division direction. Note that the processing of step S400 will be described in detail later, with reference to
In step S401, an RDDMAC 221 with which the data transfer unit 105 is equipped reads out an input image 241 of the current frame that is to undergo vector detection from the DRAM 107 via the bus 115. The amount of data that is read out depends on the size of the feature point extraction grid squares 302 and the peripheral grid squares 301 set as shown in
In step S402, the template matching image generation unit 201 generates a template matching image to be used in template matching at the time of vector detection, and outputs the generated template matching image to a WRDMAC 231 with which the data transfer unit 105 is equipped. The template matching image generation unit 201 includes a bandpass filter circuit, and cuts the high frequency component and low frequency component of image signals that is not required in the template matching processing.
In step S403, the WRDMAC 231 with which the data transfer unit 105 is equipped writes the input template matching image (hereinafter, referred to as template matching image 242) to the DRAM 107 via the bus 115. Also, the template matching image (hereinafter, referred to as template matching image 243) generated in the previous frame is stored in the DRAM 107.
In step S404, the new feature point calculation unit 202 calculates a new feature point (new tracking point) of the current frame. Note that the processing of step S404 can be executed in parallel with the processing of step S402 and step S403.
Here, the new feature point calculation unit 202 will be described in detail, with reference to
The feature filter unit 501 is constituted by a plurality of filters, such as a bandpass filter, a horizontal differential filter, a vertical differential filter and a smoothing filter. In the present embodiment, the bandpass filter cuts the unnecessary high frequency component and low frequency component of the input image (units of grid squares). Thereafter, a signal obtained by horizontal differential filtering processing using the horizontal differential filter and a signal obtained by vertical differential filtering processing using the vertical differential filter are generated. Smoothing filtering processing is then performed using the smoothing filter on signals that have undergone the horizontal and vertical differential filtering processing. The feature filter unit 501 respectively outputs the signals thus obtained as horizontal and vertical feature images.
The feature evaluation unit 502 calculates, for each pixel, with respect to the feature image (units of grid squares) obtained through filtering processing by the feature filter unit 501, a point at which the differential values around the pixel are large in multiple directions, such as a point of intersection of two edges or a point on a curve where the curvature is maximum, as a feature value using a feature evaluation equation. Hereinafter, calculation of feature values will be described, taking the Shi and Tomasi method as an example. The feature evaluation unit 502 creates an autocorrelation matrix H shown in Equation 1 from results obtained by performing horizontal differential filtering and vertical differential filtering.
In Equation 1, Ix represents the result of performing horizontal differential filtering and Iy represents the result of performing vertical differential filtering, and a computation for convolving a Gaussian filter G is performed. A Shi and Tomasi feature evaluation equation is shown in Equation 2.
Shi and Tomasi=min(λ1,λ2) (2)
Equation 2 indicates taking the smaller of the eigenvalues λ1 and λ2 of the autocorrelation matrix H of Equation 1 as the feature value.
The feature point determination unit 503 determines (selects), for each grid square (feature point selection area), a pixel having the largest feature value Pc among the feature values calculated for each pixel by the feature evaluation unit 502, as a feature point. In the present embodiment, the coordinates of feature point are represented by relative coordinates (PX,PY) whose origin (0,0) is the upper left end of the grid squares, but may be represented with absolute coordinates in an image signal.
Note that the coordinates (PX,PY) and the feature value Pc of the calculated new feature point are stored in a memory with which the new feature point calculation unit 202 is equipped. The memory has the capacity for storing the coordinates (PX,PY) and the feature value Pc of feature points of the previous frame and the current frame. In a template matching processing unit 203, a new feature point 251 is output to the tracking destination feature point determination unit 205 together with the feature value Pc, when template matching processing of the corresponding grid square is started.
In step S405, the RDDMAC 222 reads out a rectangular area of set size, centered at a tracking destination feature point 257 calculated in the previous frame, from the template matching image 242 of the current frame and a template matching image 243 of the previous frame. The tracking destination feature point 257 is input from the tracking destination feature point determination unit 205. When initially performing template matching processing, however, the tracking destination feature point has not been calculated in the previous frame, and thus the new feature point 251 of the previous frame is used instead of the tracking destination feature point 257.
Note that the rectangular area that is read out from the template matching image 242 of the current frame corresponds to the search area 305, and the rectangular area read out from the template matching image 243 of the previous frame corresponds to the template area 304. The image of the rectangular area corresponding to the search area 305 and the image of the rectangular area corresponding to the template area 304 are respectively output to the template matching processing unit 203 as a search area image 253 and a template area image 254.
In step S406, the template matching processing unit 203 calculates a correlation value using the search area image 253 and the template area image 254, and calculates a vector value based on that correlation value. In the present embodiment, the sum of absolute difference (SAD) shown in Equation 3 can be used as a method of calculating the correlation value.
S_SAD=ΣiΣj|f(i,j)−g(i,j)| (3)
In Equation 3, f(i,j) represents the pixel value at the coordinates (i,j) within the template area image 254, and g(i,j) represents the corresponding pixel value within the area that is to undergo correlation value calculation in the search area image 253. The correlation value calculation area is the same size as the template area image 254. With SAD, the correlation value S_SAD can then be obtained by calculating the absolute difference value for each pixel value f(i,j) and g(i,j) within both blocks, and deriving the sum thereof. Accordingly, the difference in luminance value between both blocks decreases as the value of the correlation value S_SAD decreases, which indicates, in other words, that the texture is similar within the template area image 254 and the correlation value calculation area.
Note that although, in the present embodiment, SAD is used as an example of a correlation value, the present invention is not limited thereto, and another correlation value such as the sum of squared difference (SSD) or normalized cross-correlation (NCC) may be used. The template matching processing unit 203 calculates a vector value of the tracking destination feature point of the previous frame, based on the position of the minimum correlation value. The template matching processing unit 203 outputs vector information 255 (calculated vector value and correlation value) to an accuracy determination unit 204.
In step S407, the accuracy determination unit 204 calculates the maximum, minimum, average and local minimum correlation values based on the correlation values calculated in step S406, and performs low contrast determination, maximum pixel value projection determination and repetition pattern determination.
In
In the low contrast determination, if the difference between the maximum and minimum correlation values within the correlation value calculation area is smaller than a threshold value set in advance, it is determined that the correlation value calculation area is low contrast. In the maximum pixel value projection determination, the prominence of the minimum correlation value within the correlation value calculation area is determined. If the result of dividing the difference between the maximum and average pixel values by the difference between the maximum and minimum pixel values is smaller than a threshold value set in advance, it is determined that the correlation value calculation area is a low peak. If the division result is larger than the threshold value, it is determined that the correlation value calculation area is a high peak. In the repetition pattern determination, if the difference between the minimum and local minimum pixel values within the correlation value calculation area is smaller than a threshold value set in advance, it is determined that the correlation value calculation area is a repeating pattern.
In step S408, the accuracy determination unit 204 outputs vector information 256 including the determination results of the low contrast determination, the maximum pixel value projection determination and the repetition pattern determination to an SRAM 206. The vector information 256 is stored in the SRAM 206 as vector data 244.
In step S409, the tracking destination feature point determination unit 205 calculates the tracking destination feature point of the current frame that is for use in template matching processing of the next frame, based on the tracking destination feature point 257 and the vector data 244 of the previous frame. The tracking destination feature point calculated here is used as the tracking destination feature point 257 of the previous frame, when template matching processing is next executed in step S405. Note that the processing of step S409 will be described in detail later, with reference to
As a result of the above processing, template matching processing of the current frame (i.e., processing for tracking the feature points of the previous frame in the current frame) is completed. Thereafter, when the shot image of the next frame is input to the image processing unit 104, the processing of steps S401 to S409 is executed again, with the image of the next frame as the image of the current frame. Accordingly, in the present embodiment, processing for tracking feature points across one or more shot images that are shot after a predetermined shot image that serves as a reference (e.g., first shot image) is performed. Also, the feature point tracking processing of
Next, the processing of step S409 will be described in detail, with reference to
In step S702, the tracking destination feature point determination unit 205 determines whether the tracking destination feature point calculated in step S701 is valid (whether the calculated tracking destination feature point can be used in template matching processing of the next frame), based on the determination result (determination result of step S407) that is included in the vector data 244. If it is determined that the correlation value calculation area is low contrast by the low contrast determination, it is determined that the tracking destination feature point is not valid. Also, if it is determined that the correlation value calculation area is a low peak in the maximum projection determination, and, if it is also determined that the correlation value calculation area is a repetition pattern in the repetition pattern determination, it is determined that the tracking destination feature point is not valid. In other cases, it is determined that the tracking destination feature point is valid. If the tracking destination feature point is valid, the processing advances to step S703, and, if the tracking destination feature point is not valid, the processing advances to step S704.
In step S703, the tracking destination feature point determination unit 205 determines whether the tracking destination feature point calculated in step S701 is included inside the processing area. If the tracking destination feature point calculated in step S701 is included inside the processing area, the processing advances to step S705, and, if this is not the case, the processing advances to step S704. Note that, in the case where the correlation value calculation area is always set inside the processing area in the template matching processing (step S406 of
In step S704, the tracking destination feature point determination unit 205 replaces the tracking destination feature point calculated in step S701 with a new feature point of the current frame calculated in step S404 of
In step S705, the tracking destination feature point determination unit 205 determines whether all of the tracking destination feature points have been processed. If all of the tracking destination feature points have been processed, the processing of this flowchart is completed. If any unprocessed tracking destination feature points remain, the processing from step S701 onward is similarly performed for the next unprocessed tracking destination feature point.
Next, the processing of step S400 will be described in detail, with reference to
The division method for dividing the processing area horizontally when the digital camera 100 is held horizontally, as shown in
In step S1002, the CPU 112 determines whether the digital camera 100 is horizontally positioned based on information from the gyro sensor 116. In the case of being horizontally positioned, the processing advances to step S1003, and, if this is not the case, the processing advances to step S1004. Note that, in cases such as where the digital camera 100 is oriented diagonally, the CPU 112 determines whether the orientation of the digital camera 100 is close to landscape orientation or is close to portrait orientation, and, if close to landscape orientation, determines that the digital camera 100 is horizontally positioned.
In step S1003, the CPU 112 determines whether the movement direction of the object is horizontal based on information from the object detection unit 123. In the case of horizontal movement, the processing advances to step S1005, and, if this is not the case, the processing advances to step S1006. Note that, in the case where the movement direction of the object is diagonal, the CPU 112 determines whether the horizontal movement component is greater or the vertical movement component is greater, and, if the horizontal movement component is greater, determines that the movement direction of the object is horizontal.
In step S1004, the CPU 112 determines whether the movement direction of the object is horizontal based on information from the object detection unit 123. This determination processing is substantially the same as the determination processing of step S1003, and the movement direction of the object being horizontal or vertical is determined on the basis of the direction of gravity.
In step S1005, the CPU 112 sets the processing areas by division method A, by controlling the division direction setting unit 260 of the motion vector detection units 121 and 122. For example,
In step S1006, the CPU 112 sets the processing areas by division method B, by controlling the division direction setting unit 260 of the motion vector detection units 121 and 122. For example,
After the processing of step S1005 or step S1006 is performed, the motion vector detection units 121 and 122 independently execute the processing from step S401 of
According to the first embodiment, as described above, the digital camera 100 determines the movement direction of the object, and sets a plurality of processing areas that are arranged in a different direction from the movement direction of the object, within the shooting range. It thereby becomes possible to reduce the possibility of feature points moving in the direction of another processing area and tracking failure (loss of feature points) occurring.
Note that, in the above description, the digital camera 100 is given as determining the division method based on position information of the digital camera 100 (whether the digital camera 100 is horizontally positioned or vertically positioned) and movement information of the object (whether the object is moving in the horizontal direction or the vertical direction). However, the digital camera 100 may determine the division method with reference to movement direction information of the digital camera 100, information obtained by averaging the movement direction of a plurality of feature points, vector information obtained by averaging a plurality of motion vectors, and the like.
In the case where the movement direction information of the digital camera 100 is the horizontally direction, the possibility that the movement direction of the feature point will also be the horizontal direction increases. Thus, the digital camera 100 selects division method A if the digital camera 100 is horizontally positioned and division method B if the digital camera 100 is vertically positioned. Also, in the case where movement direction information of the digital camera 100 is the vertical direction, the possibility that the movement direction of the feature point will also be the vertical direction increases. Thus, the digital camera 100 selects division method B if the digital camera 100 is horizontally positioned and division method A if the digital camera 100 is vertically positioned. In the case where the movement direction of the digital camera 100 is diagonal, the digital camera 100 determines the movement direction to be the horizontal direction if the movement component of the horizontal direction is greater than the movement component of the vertical direction.
In the case where the information obtained by averaging the movement direction of a plurality of feature points is the horizontal direction, the digital camera 100 selects division method A if the digital camera 100 is horizontally positioned and division method B if the digital camera 100 is vertically positioned.
In the case where the information obtained by averaging the movement direction of a plurality of feature points is the vertical direction, the digital camera 100 selects division method B if the digital camera 100 is horizontally positioned and division method A if the digital camera 100 is vertically positioned. In the case where the information obtained by averaging the movement direction of a plurality of feature points is the diagonal direction, the digital camera 100 determines the movement direction to be the horizontal direction if the movement component of the horizontal direction is greater than the movement component of the vertical direction.
In the case where the vector information obtained by averaging a plurality of motion vectors is the horizontal direction, the possibility that the movement direction of the feature point will also be the horizontal direction increases. Thus, the digital camera 100 selects division method A if the digital camera 100 is horizontally positioned and division method B if the digital camera 100 is vertically positioned. In the case where the vector information obtained by averaging a plurality of motion vectors is the vertical direction, the possibility that the movement direction of the feature point will also be the vertical direction increases. Thus, the digital camera 100 selects division method B if the digital camera 100 is horizontally positioned and division method A if the digital camera 100 is vertically positioned. In the case where the vector information obtained by averaging a plurality of motion vectors is the diagonal direction, the digital camera 100 determines the movement direction to be the horizontal direction, if the movement component of the horizontal direction is greater than the movement component of the vertical direction.
Second EmbodimentA second embodiment describes an other example of the processing for determining the division direction of the processing area (step S400 of
In
Next, when input of the lower half of the input image 241 to the DRAM 107 starts, in the motion vector detection unit 122 which processes the lower half of the image, similar processing to the motion vector detection unit 121 is performed. In the motion vector detection unit 122, the RDDMAC 222 reads out an image area centered on the feature points 1121 to 1132, and template matching processing is implemented during periods 13 to 24 enclosed with broken lines.
Note that the time required for template matching processing becomes longer as the search area becomes larger, since the number of iterations of matching increases. Thus, there are cases where template matching processing is not completed within the 1V period as shown in
On the other hand, in the case of division method B, as shown in
In step S1301, the CPU 112 recognizes the movement amount of the object based on information from the object detection unit 123, and determines the search area size of template matching processing based on the movement amount of the object. Since the motion vector value increases as the movement amount of the object increases, search area size also increases in order to secure the required search range.
In step S1302, the CPU 112 calculates the time required for template matching processing in the case of division method A (in the case where a plurality of processing areas are arranged in the up-down direction of the image sensor 102). In
In step S1303, the CPU 112 determines whether the template matching processing can be completed within the 1V period (within one frame period of moving image shooting), based on the required time calculated in step S1302. In the case where the template matching processing can be completed within the 1V period, the processing advances to step S1005, and, if this is not the case, the processing advances to step S1006.
According to the second embodiment, as described above, the digital camera 100 determines whether the template matching processing can be completed within the 1V period in division method A. In the case where the template matching processing cannot be completed within the 1V period in division method A, the digital camera 100 selects division method B in which the plurality of processing areas are arranged in the right-left direction of the image sensor 102, regardless of the movement direction of the object. It thereby becomes possible to reduce the possibility of not being able to complete the template matching processing within the 1V period.
Note that, in the case where the search area size is large, there is a possibility of not being able to complete template matching processing within the 1V period even with division method B. In this case, the digital camera 100 completes the template matching processing within the 1V period, by omitting the template matching processing in period 12 and period 24 of
A third embodiment describes a variation of the feature point supplementation processing (step S704 of
In step S1601, the tracking destination feature point determination unit 205 replaces the tracking destination feature point calculated in step S701 with a new feature point of the current frame calculated in step S404 of
Note that, in the third embodiment, the plurality of processing areas that are set do not necessarily need to be arranged in a different direction from the movement direction of the object.
Also, in the above description, the digital camera 100 is given as being provided with two motion vector detection units. However, the present embodiment is also applicable in the case where the digital camera 100 is provided with three or more motion vector detection units.
Also, the digital camera 100 may calculate an object area (size) using the object detection unit 123, and set supplementation priority grid squares limited to grid squares that the object will enter, among the feature point extraction grid squares that are next to or at least partially overlap with another processing area.
Also, in the above description, the digital camera 100 is given as performing motion vector detection on the entire screen. However, the digital camera 100 may perform motion vector detection, limited to the object area detected by the object detection unit 123.
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2018-062258, filed Mar. 28, 2018, which is hereby incorporated by reference herein in its entirety.
Claims
1. An image processing apparatus comprising:
- a determination unit configured to determine a movement direction of an object;
- a setting unit configured to set, within a shooting range, a plurality of processing areas that are arranged in a different direction from the movement direction;
- a selection unit configured to select a tracking point in each processing area of a predetermined shot image; and
- a tracking unit configured to track, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image.
2. The image processing apparatus according to claim 1,
- wherein the setting unit sets a plurality of selection areas within each processing area, and
- the selection unit selects the tracking point in each selection area of each processing area of the predetermined shot image.
3. The image processing apparatus according to claim 2,
- wherein, in a case where, in a specific shot image of the one or more shot images, the tracking unit fails in tracking of a specific tracking point selected in a specific selection area of the predetermined shot image, the selection unit selects a new tracking point to be substituted for the specific tracking point in the specific selection area of the specific shot image.
4. The image processing apparatus according to claim 2,
- wherein, in a case where, in a specific shot image of the one or more shot images, the tracking unit fails in tracking of a specific tracking point selected in a specific processing area of the predetermined shot image, the selection unit selects a new tracking point to be substituted for the specific tracking point in a specific selection area among the plurality of selection areas of the specific processing area of the specific shot image, and
- the specific selection area is one of one or more selection areas that are next to or at least partially overlap with another processing area.
5. The image processing apparatus according to claim 4,
- wherein in a case where the object is moving in a direction of entering the specific processing area of the specific shot image, the specific selection area is a selection area in a position that the object will enter, among the one or more selection areas that are next to or at least partially overlap with another processing area.
6. The image processing apparatus according to claim 2,
- wherein the setting unit sets the plurality of selection areas in a position where the object exists within each processing area.
7. The image processing apparatus according to claim 1,
- wherein the predetermined shot image and the one or more shot images are generated by moving image shooting that uses an image sensor, and
- the setting unit: determines whether tracking of the tracking point by the tracking unit in one shot image in a case where the plurality of processing areas are arranged in an up-down direction of the image sensor will be completed within a one frame period of the moving image shooting; and in a case where tracking of the tracking point will not be completed within the one frame period, sets the plurality of processing areas so as to be arranged in a left-right direction of the image sensor, regardless of the movement direction.
8. An image processing apparatus comprising:
- a setting unit configured to set a plurality of processing areas within a shooting range, and to set a plurality of selection areas within each processing area;
- a selection unit configured to select a tracking point in each selection area of each processing area of a predetermined shot image; and
- a tracking unit configured to track, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image,
- wherein, in a case where, in a specific shot image of the one or more shot images, the tracking unit fails in tracking of a specific tracking point selected in a specific processing area of the predetermined shot image, the selection unit selects a new tracking point to be substituted for the specific tracking point in a specific selection area among the plurality of selection areas of the specific processing area of the specific shot image, and
- the specific selection area is one of one or more selection areas that are next to or at least partially overlap with another processing area.
9. An image capturing apparatus comprising:
- the image processing apparatus according to claim 1; and
- an image capturing unit configured to generate the predetermined shot image and the one or more shot images.
10. An image capturing apparatus comprising:
- the image processing apparatus according to claim 8; and
- an image capturing unit configured to generate the predetermined shot image and the one or more shot images.
11. An image processing method comprising:
- determining a movement direction of an object;
- setting, within a shooting range, a plurality of processing areas that are arranged in a different direction from the movement direction;
- selecting a tracking point in each processing area of a predetermined shot image; and
- tracking, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image.
12. An image processing method comprising:
- setting a plurality of processing areas within a shooting range, and to set a plurality of selection areas within each processing area;
- selecting a tracking point in each selection area of each processing area of a predetermined shot image; and
- tracking, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image,
- wherein, in a case where, in a specific shot image of the one or more shot images, tracking of a specific tracking point selected in a specific processing area of the predetermined shot image fails, selecting a new tracking point to be substituted for the specific tracking point in a specific selection area among the plurality of selection areas of the specific processing area of the specific shot image, and
- the specific selection area is one of one or more selection areas that are next to or at least partially overlap with another processing area.
13. A non-transitory computer-readable storage medium which stores a program for causing a computer to execute an image processing method comprising:
- determining a movement direction of an object;
- setting, within a shooting range, a plurality of processing areas that are arranged in a different direction from the movement direction;
- selecting a tracking point in each processing area of a predetermined shot image; and
- tracking, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image.
14. A non-transitory computer-readable storage medium which stores a program for causing a computer to execute an image processing method comprising:
- setting a plurality of processing areas within a shooting range, and to set a plurality of selection areas within each processing area;
- selecting a tracking point in each selection area of each processing area of a predetermined shot image; and
- tracking, inside each processing area, the tracking point across one or more shot images that are shot after the predetermined shot image,
- wherein, in a case where, in a specific shot image of the one or more shot images, tracking of a specific tracking point selected in a specific processing area of the predetermined shot image fails, selecting a new tracking point to be substituted for the specific tracking point in a specific selection area among the plurality of selection areas of the specific processing area of the specific shot image, and
- the specific selection area is one of one or more selection areas that are next to or at least partially overlap with another processing area.
Type: Application
Filed: Mar 27, 2019
Publication Date: Oct 3, 2019
Patent Grant number: 11153479
Inventor: Hiroyuki Yaguchi (Ichikawa-shi)
Application Number: 16/366,107