Image processing method for detecting objects using relative motion
An image based obstacle detection method. A camera mounted on a vehicle provides a set of image frames while the vehicle is in motion. The image frames define an image plane having a vertical aspect and a horizontal aspect. The relevancy of an object is determined by (i) selecting first and second feature points from the object that are spaced apart vertically in a first image frame; (ii) tracking the positions of the first and second feature points over at least a second image frame; and (iii) deciding the object to be relevant if the first and second feature points move dissimilar distances in physical space, within a tolerance, and deciding the object to be irrelevant otherwise. The motion of relevant objects is then estimated to determine if any relevant object is likely to become an obstacle to the vehicle.
Latest MAGNA ELECTRONICS INC. Patents:
- Vehicular autonomous control system based on learned and predicted vehicle motion
- Automotive radar / LIDAR behind reflective surfaces
- Vehicular driver monitoring system with camera view optimization
- Vehicular central monitoring system with central server
- Vehicular multi-sensor system using a camera and lidar sensor to detect objects
The present application is a 371 U.S. national phase application of PCT Application No. PCT/CA2012/000057 filed Jan. 20, 2012 which claims priority to U.S. Provisional Patent Application No. 61/478,717, filed Apr. 25, 2011.
FIELD OF INVENTIONThe invention relates to the field of image processing, and more particularly to a method and system for discerning three-dimensional objects using relative motion between the camera and the objects, and an object detection system based on same.
BACKGROUND OF INVENTIONMany vehicles now incorporate one or more obstacle detection system which use cameras to provide a field of view of the roadway and surrounding environment. The obstacle detection systems process the video stream or sequence of image frames provided by the camera in an attempt to ascertain the existence of an obstacle in the path of vehicle. Obstacle detection algorithms do this by attempting identify a variety of objects localized in one or more predefined regions of interest in the captured field of view, determine whether or not such objects are real, and estimate their path relative to the vehicle to determine whether or not such objects will indeed become obstacles.
The conventional approach to solving such a problem is to develop a series of classifiers and detectors to detect a group of predefined objects. The classifiers work in a cascade or in a parallel fashion to detect image content such as pedestrians, vehicles and geometrically shaped objects such as balls and poles. Such systems may require significant computational resources provided by expensive signal processors. In addition, in such a system, all the objects that need to be detected have to be defined in the development stage and require extensive training for the classifiers. Such a process is expensive and time consuming and might not be generic enough with respect to all possible different objects that might be encountered in the real world.
SUMMARY OF INVENTIONIn order to overcome some of the above mentioned shortcomings, the invention proposes a more generic approach which does not depend upon the design of specific classifiers. Instead, the invention uses the relative motion between the vehicle and the surrounding environment to identify whether or not detected objects are relevant. This approach uses a single image sensor and can be employed for detecting generic objects.
Generally speaking, the invention utilizes the principle that spaced apart physical points that in reality have a vertical dimension will move at different rates when projected to the image plane provided by the vehicular camera, whereas spaced apart physical points that in reality lie only horizontally along the ground plane will move at the same rate when projected to the image plane. Using this principle the invention is able to rapidly distinguish between three dimensional objects that could be relevant to the obstacle detection analysis and irrelevant two dimensional objects that, for example, may be pavement markings or artifacts such as shadows.
The foregoing principle will be better understood with reference to
However, referring additionally to
Now consider the vertical object 30. It has two opposing edge points that lie at initial positions P1, P2. The point P1, lying along ray 28, projects onto the ground plane at point G1 as previously discussed. Point P2 in this example lies along the ground plane (but it need not). Let the vertical object 30 move relative to the camera by a real distance d. The top edge point moves from position P1 to position P1′, and the bottom edge point moves from position P2 to P2′. Now calculate the distance moved in the ground plane by these end points. P1 projects onto the ground plane at point G1. Point P1′ projects onto the ground plane at point X. So, a distance calculation in the ground plane between points P1 and P1′ effectively yields a result computing the distance between G1 and X, resulting in a much larger distance d++. On the other hand, P2 and P2′ are located on the ground plane so the distance between them will be computed as d. The change d++ in the ground plane between P1 and P1′ differs considerably from the change d between P2 and P2′, thus enabling a vertical object to be distinguished from an object lying substantially along the ground plane.
The principle holds for distinguishing between an object lying substantially along any selected real horizontal plane and an object lying generally transverse to the selected horizontal plane. Likewise, the principle applies even when the vertical object does not touch the selected real horizontal plane because vertically spaced apart points will move at different rates when projected onto the selected horizontal plane.
In the preferred obstacle detection system described in greater detail below, a vehicular camera is utilized to capture a video stream or sequence of image frames. The image frames are processed by an image processor which executes an obstacle detection algorithm. The preferred system then processes the images to extract edges therein. (An example of a road image 10 and its edge image 12 can be seen in
The preferred system processes the edge images to identify potential objects preferably within a region of interest (ROI) in the image frame. Objects (obstacles or not) are identified by their boundaries or contours. If an object moves across successive image frames, the edges of such an object will also move and so edge based features are used to identify the location of the objects and are also used to track the objects across frames. However, owing to environmental effects and low contrast, it is possible that the edges of an object are not always isolated from other objects or the general background. In order to make the object localization more reliable, the preferred system examines different features such as color and texture cues, in conjunction with the edges, in order to detect and isolate objects from one another.
The isolated objects could be actual three dimensional objects or they could be two dimensional objects on the ground such as shadows or pavement markings. Using a single monocular image sensor, it can be quite difficult to differentiate one type of object from another. However, if there is relative motion between these objects and the camera, then using the principles outlined above the preferred system can estimate which of the detected objects are above a certain specified height threshold and which detected objects are not (such as markings on the ground). Accordingly, the preferred system examines the motion of at least two different feature points in the edge image of each object across image frames. If there is relative motion between the camera and the object, for the same physical distances traveled, points lying at different physical heights will have different rates of change in the image plane (termed “differential-gradient effect”). Thus, if the selected feature points in the edge image of an isolated object do not have different rates of change, within a tolerance, the preferred system deems that object to be an irrelevant two dimensional object. Similarly, if the selected feature points in the edge image of an isolated object do have different rates of change, within a tolerance, the preferred system deems that object to be relevant and estimates the relative path of that object to determine whether or not it will become an obstacle to the vehicle.
Thus, one aspect of the invention provides a method of obstacle detection. In this method, a vehicle is provisioned with a camera having an external field of view and a set of image frames provided by the camera are acquired whilst the vehicle is in motion. The image frames define an image plane having a vertical aspect and a horizontal aspect. The relevancy of an object is determined by (i) selecting first and second feature points from the object that are spaced apart vertically in a first image frame; (ii) tracking the positions of the first and second feature points over at least a second image frame; and (iii) deciding the object to be relevant if the first and second feature points move dissimilar distances in physical space, within a tolerance, and deciding the object to be irrelevant otherwise. The motion of relevant objects is then estimated to determine if any relevant object is likely to become an obstacle to the vehicle.
In practice, an edge detection is preferably carried out on the set of images frames and one or more objects are isolated based on their edge contours. The most desirable feature points for tracking across frames are the topmost and bottommost pixel points of any given isolated object as these can be used to estimate the relative physical displacement of the object. For this estimated displacement of the object, by comparing the estimated pixel movement of the topmost detected point of the object with the actual pixel movement of the same point, the object can be determined to be a relevant object.
In addition, in practice, a given pixel is preferably position is de-warped to account for lens distortion prior to determining a physical location corresponding to the given pixel position.
The foregoing and other aspects of the invention will be better understood with respect to the attached drawings, wherein:
For reference purposes, the following mathematical nomenclature is used in this description. A vector or a matrix is represented in bold face, while a scalar entity is represented in a regular face. The lower case is used to denote the dimensions and pixel locations in two dimensional image space. Thus, the collection of all the pixels in an image can be denoted as f and a pixel at column x and row y in the image f can be represented as f(x, y). The coordinates of a point in the three-dimensional world are represented using the upper case. A point on the ground can be represented in its vector form as X, or also as [Xw, Yw, Zw].
The system 100 outputs one or more detected objects 106 (if any). The detected objects 106 are output as an array of data structures indicating: object centroid; object bounding box (position and breadth); and an estimated distance from the vehicle.
A differential, gradient-based, segmentation module 110 receives the input information 102-105. At an initial stage 112 the module 110 uses the input image frames to hypothesize or postulate the existence of an object by detecting and isolating objects from the background, within a predefined region of interest (ROI). At a subsequent stage 114 the module 110 filters the postulated objects to discern those which have dimensions above a defined height threshold. This is an important practical consideration because most real life objects that pose any threat to the vehicle (or to a pedestrian) have a significant vertical component. This module thus generates initial object hypotheses at output 116 which can be further refined and processed for final output.
The segmentation module 110 is based on a differential gradient based pseudo-stereo effect, which has the following features:
-
- Points lying on the same vertical axis at different planar heights (in reality) move at different pixel rates in the captured video. This allows for separation of hypothesized objects which lie on the ground plane (or other selected horizontal plane) versus those which are three dimensional in nature and pose a threat to the vehicle.
- Owing to the above principle, this approach allows the detection of even those objects which are stationary around the vehicle if the vehicle itself is moving. This allows for detection of stationary pedestrians or poles or other objects which the vehicle might be driving towards.
- Only relative motion is required between the objects and vehicle.
Referring additionally to
The edge detection block 110A uses the contours of objects as a means to segment them from the background image. An object can be assumed to be bound by its contour or its edge, which can be extracted using the gradient operator.
The edge points of an image can be detected by finding the zero crossings of the second derivative of the image intensity. The idea is illustrated for a one dimensional signal f(1) in
There are three main steps in Laplacian of Gaussian edge detection: filtering, enhancement, and detection.
Gaussian filtering is used for smoothing and the second derivative of which is used for the enhancement step. The detection criterion is the presence of a zero crossing in the second derivative with a corresponding large peak in the first derivative.
In this approach, noise is first reduced by convoluting the image with a Gaussian filter which removes isolated noise points and small structures. Those pixels, that have locally maximum gradient in the zero crossing of the second derivative, are considered as edges by the edge detector. To avoid detection of insignificant edges, only the zero crossings whose corresponding first derivative is above some threshold, are selected as edge point. The edge direction is obtained using the direction in which zero crossing occurs.
The output of the LoG operator; h(x,y); is obtained by the convolution operation:
where, Δ and * are the gradient and convolution operators, respectively. σ is the standard deviation of the smoothing function g( ), x and y are the location column and the row index, respectively, and f(x,y) represents the pixel at column x and row y in the input image
Once the objects have been isolated using their contours, each object or group of objects can be identified by its edge. The feature selection and tracking block 110B assumes that the bottom of the each objects lies on the ground while the top of the object is either on the ground or not, depending if the object is flat or has a three-dimensional shape. Most preferably the top and bottom of each contour are selected as the points of interest in the image frame which need to be investigated to determine if the object is above a certain threshold of height or not. These topmost and bottommost points of each edge contour are then tracked from the current frame to the next frame to find their corresponding location in the next frame. For example,
In addition, for more robustness, instead of just selecting the topmost and bottommost points, the feature selection and tracking block 110B preferably also selects and processes a set of additional feature points along the contour.
Once the feature points that define the top and bottom of each object are isolated, they, as well as any other selected feature points, can be tracked across frames using a block matching algorithm. (Note that there is no need to track the entire image or object contour, only the selected feature points need to be tracked across frames.)
Any given feature point will likely subsist for least a few following image frames and with the typical frame rate of 30 fps the given feature point will move relatively slowly across image frames. The block matching algorithm establishes a macro block around the feature point and looks for a best match. For example,
The distance estimation block 110C calculates the physical distances to the selected feature points in the ground plane.
A point in the image, xi, can be defined as
xi=[xpyp1]T
It is desired to compute the real world coordinates of this point on the ground plane, given the camera position and geometry. Before estimating the distance, a selected pixel location is de-warped using the distortion function of the lens, which is one of the intrinsic parameters provided in input 105. In a de-warped image, every image point can be expressed as a 3D Plucker line matrix, which correlates to one of the rays running through the camera focal point and impinging upon the image plane (see
Xi=H·xi
where Xi[X1 Y1 Z1 N1]T
The real world coordinates of the image point on the ground plane can be estimated with respect to the origin using Xi as: [Xw=X1/N1; Yw=Y1/Ni; Zw=Z1/N1]. To compute the real world distance of the various points using the above equation, the intrinsic and the extrinsic parameters of the camera being used are necessary. Given these parameters, the Homography matrix, H, can be estimated as the inverse of the projection matrix, P, which is defined as:
In the above equations, f and p are the focal length of the lens and the pixel size of the sensor, respectively. axisX and axisY are the column and row coordinates of the optical center on the image plane. α, β, and γ represent the angles of rotation around X, Y, and Z axes, respectively, and t1, t2, and t3 are the translations along X, Y, and Z axes, respectively.
The object detection block 110D filters the segmented objects based on the estimated real height of the objects. Those objects that are estimated to have a real height greater than a selected threshold are considered to be relevant objects worth tracking, whereas those objects that are estimated to have a real height less than a selected threshold are considered to be irrelevant objects not worth tracking.
For height threshold verification, the object detection block 110D assumes that there is some relative motion between the object and the camera, i.e., either the camera moves or the object itself moves with respect to the camera. For example,
The steps involved in this detection are as follows:
(1) Estimate the initial distance of the object from the reference camera using the pixel location of the bottommost point, O1b, and use it with the Homography matrix to estimate the real-world coordinate, D1, in the ground plane. See, for example,
(2) Track the motion of the bottommost point the object from pixel location O1b to pixel location O2b and using the Homography equation estimate the real-world coordinate D2 of the bottommost point on the ground plane, and the distance ΔD the bottommost point has moved. See, for example,
(3) Use the Homography equation to estimate a real world coordinate of the topmost point of the object corresponding to its location O1t in the first frame. Assume that the top of the object is at a height T, which will correspond to a point DT at this height. See
(4) Since the top and the bottom points of the physical object move by the same distance in the real world, then presuming that the topmost point of the object it is at height T, the new real world location of the top-most point, D*T, is DT+ΔD. See
(5) Using the inverse of the Homography equation, compute the pixel coordinate O*2T of the top-most point after accounting for its real world displacement if it was at a height T. See
(6) Compare the pixel displacement of the estimated pixel location of the top most point with respect to the initial location, ∥O2t*−O1t∥, with that of actual pixel location of the tracked topmost point of the object with respect to the initial location ∥O2t−O1t. If ∥O2t*−O1f∥<∥O2t−O1t∥, then the topmost point of the object is above the height threshold and if ∥O2t*−O1t∥>∥O2t−O1t∥, then the topmost point of the object is below the height threshold. See, for example,
The foregoing steps utilized the image plane. It should also be appreciated that the analysis can just as well be carried out in real physical space. In this methodology, as exemplified in
It will also be understood that in the case of T=0 the selected horizontal physical plane corresponds to the ground plane.
Referring back to
Stage 122—temporal hypotheses assignment—tracks postulated objects over time. The postulated objects from stage 118 are further evaluated using the predicted hypotheses from the previous image frame. The corresponding hypotheses are assigned to the current frame where a match is found. However, if no match is found between the predicted hypothesis and the current estimated hypothesis, then a new hypothesis is instantiated.
In addition, if a hypothesis existed in the previous frame but is not currently detected, then this hypothesis is not immediately rejected, but is held in memory for a specified number of frames to verify that the postulated object is no longer present. In case where the postulated object is actually present, the detection may be missed intermittently, and therefore this temporal processing leads to a significant reduction in the false negatives.
Stage 126 is a distance and trajectory estimation block. Before the physical distance of the pixel locations can be estimated at step 128, the fish-eye distortion of the lens needs to be corrected. See
The de-warped points used for estimating the distance of postulated objects are preferably the bottommost points of the objects which are assumed to lie on the ground plane. These points can be used to estimate the distance of the objects from the camera and this distance can then be translated to the vehicle coordinates. This allows the further processing of relevant postulated objects based on their relative distance to the vehicle.
The distance estimation of the same postulated object across multiple frames is subject to small variations owing to real world, non-ideal conditions. Thus, these estimated distances are filtered using a Kalman filter which can smooth out instantaneous variations. The same filter can also be used to predict the location of the object in the next frame and thus help predict the trajectory of the object, which is computed at step 132 (
Owing to the nature of the problem, a modified Kalman filter is preferably used, which has the following order of steps:
-
- Update Step: Update the filter parameters and estimate the filtered distance of the object detected in the current frame, ({circumflex over (X)},Ŷ), using the input real world distance (X,Y). (See
FIG. 12 .) - Prediction Step: Predict the distance of the object in the next frame.
- Update Step: Update the filter parameters and estimate the filtered distance of the object detected in the current frame, ({circumflex over (X)},Ŷ), using the input real world distance (X,Y). (See
Stage 136 provides a hypothesis prediction. Based on the predicted physical location of the object, the location of each hypothesis in the next frame is estimated by projecting the distances back to the image plane. These projected pixel locations can then be used for the temporal assignation of the hypotheses in the next frame.
The system output is provided at stage 106. Based on the trajectory estimation of each object, if the detected object is in the path of the vehicle, or poses a threat to the vehicle, then the object is output using its centroid as well as its bounding region or contour. This output may be displayed visually on a display screen or the like mounted in the vehicle and may be accompanied by audible warnings such as an alarm or voice, or by tactile feedback such as a buzz in the steering wheel. In addition, in the event the system detects an imminent crash by virtue of an obstacle being immediately in the path of the vehicle various pre-emptive actions may be taken such as automated braking or tensioning of set belts.
Those skilled in the art will appreciate that a variety of modifications may be made to the embodiments described herein without departing from the spirit of the invention.
Claims
1. An obstacle detection method, comprising:
- (a) provisioning a vehicle with a camera having an external field of view, the camera having an imager defining an image plane having a vertical aspect and a horizontal aspect, wherein the imager comprises an array of sensing pixels;
- (b) acquiring a set of image frames captured by the camera while the vehicle is in motion;
- (c) determining an object present in the set of captured image frames by edge detection;
- (d) determining a relevancy of the determined object by (i) selecting first and second feature points of the determined object that are spaced apart vertically in a first captured image frame, (ii) tracking positions of the first and second feature points of the determined object over at least a second captured image frame, and (iii) determining that the determined object is a relevant object responsive to a determination that the first and second feature points move dissimilar distances in physical space between the first and second captured image frames and within a tolerance;
- (e) estimating motion of determined relevant objects relative to the vehicle to determine if any determined relevant object is likely to become an obstacle to the vehicle;
- wherein step (d)(iii) is determined by reference to the image plane and, where the first feature point is vertically lower than the second feature point, and wherein the method includes:
- estimating, from the first captured image frame, a physical distance D1 of the first feature point O1b to the camera in a selected physical horizontal plane;
- estimating, from the first captured image frame, a physical location DT of the second feature point O1T assuming that the second feature point physically lies at a height T above the selected physical horizontal plane;
- estimating, from the second captured image frame, a physical distance D2 of the first feature point O2b to the camera in the selected physical horizontal plane, and determining the distance ΔD between D2 and D1;
- estimating a physical location D*T which is equivalent to DT+ΔD, with D*T being located at a height T above the selected physical horizontal plane;
- computing a nominal feature point coordinate O*2t in the image plane corresponding to D*T; and
- determining that the determined object is the relevant object responsive to a determination that the absolute distance between the nominal feature point coordinate O*2t and the second feature point coordinate in the first captured image frame O1T is less than the absolute distance between the second feature point coordinate in the second captured image frame O2T and the second feature point coordinate in the first captured image frame O1T.
2. A method according to claim 1, wherein step (d)(iii) is determined by reference to physical space and includes:
- determining first and second physical locations along a selected physical horizontal plane corresponding to pixel positions of the first and second feature points in the first captured image frame;
- determining third and fourth physical locations along the selected real horizontal plane corresponding to pixel positions of the first and second feature points in the second captured image frame;
- determining a first distance Δd1 between the third and first physical locations and determining a second distance Δd2 between the fourth and second physical locations; and
- determining that the determined object is the relevant object responsive to a determination that the difference between the first distance Δd1 and the second distance Δd2 exceeds a threshold, and otherwise determining that the determined object is not a relevant object.
3. A method according to claim 2, including unwarping a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.
4. A method according to claim 3, wherein the first and second feature points are portions of top and bottom edge contours of the determined object.
5. A method according to claim 1, wherein the selected physical horizontal plane is a ground plane.
6. A method according to claim 5, including unwarping a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.
7. A method according to claim 6, wherein the first and second feature points are portions of top and bottom edge contours of the determined object.
8. An obstacle detection system for a vehicle, comprising:
- a camera mounted at the vehicle, the camera capturing a set of image frames while the vehicle is in motion, the camera comprising an imager defining an image plane having a vertical aspect and a horizontal aspect, wherein the imager comprises an array of sensing pixels;
- a controller coupled to the camera, the controller configured to determine an object in the captured image frames and determine a relevancy of the determined object by (i) selecting first and second feature points on the determined object that are spaced apart vertically in a first captured image frame, (ii) tracking positions of the first and second feature points over at least a second captured image frame, and (iii) determining that the determined object is a relevant object responsive to a determination that the first and second feature points move dissimilar distances in physical space and within a tolerance, and otherwise determining that the determined object is not a relevant object; and
- the controller estimating relative motion of determined relevant objects to determine if any relevant object is likely to become an obstacle to the vehicle; and
- the controller providing an indication to a driver of the vehicle of determined obstacles;
- wherein the controller determines the relevancy of the determined object by reference to the image plane and, where the first feature point is vertically lower than the second feature point, the controller:
- estimates, from the first captured image frame, a physical distance D1 of the first feature point O1b to the camera in a selected physical horizontal plane;
- estimates, from the first captured image frame, a physical location DT of the second feature point O1T assuming that the second feature point physically lies at a height T above the selected physical horizontal plane;
- estimates, from the second captured image frame, a physical distance D2 of the first feature point O2b to the camera in the selected physical horizontal plane, and determines the distance ΔD between D2 and D1;
- estimates a physical location D*T which is equivalent to DT+ΔD, with D*T being located at a height T above the selected physical horizontal plane;
- computes a nominal feature point coordinate O*2t in the image plane corresponding to D*T; and
- determines the determined object to be the relevant object responsive to a determination that the absolute distance between the nominal feature point coordinate O*2t and the second feature point coordinate in the first captured image frame O1T is less than the absolute distance between the second feature point coordinate in the second captured image frame O2T and the second feature point coordinate in the first captured image frame O1T.
9. An obstacle detection system according to claim 8, wherein the controller determines the relevancy of the determined object by reference to physical space, and wherein the controller is configured to:
- determine first and second physical locations along a selected physical horizontal plane corresponding to pixel positions of the first and second feature points in the first captured image frame;
- determine third and fourth physical locations along the selected real horizontal plane corresponding to pixel positions of the first and second feature points in the second captured image frame;
- determine a first distance Δd1 between the third and first physical locations and determining a second distance Δd2 between the fourth and second physical locations; and
- determine that the determined object is the relevant object if the difference between the first distance Δd1 and the second distance Δd2 exceeds a threshold, and otherwise determining that the determined object is not a relevant object.
10. An obstacle detection system according to claim 9, wherein the controller de-warps a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.
11. An obstacle detection system according to claim 10, wherein the first and second feature points are portions of top and bottom edge contours of the determined object.
12. An obstacle detection system according to claim 9, wherein the selected physical horizontal plane is a ground plane.
13. An obstacle detection system according to claim 12, wherein the controller de-warps a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.
14. An obstacle detection system according to claim 13, wherein the first and second feature points are portions of top and bottom edge contours of the determined object.
4720790 | January 19, 1988 | Miki et al. |
4987357 | January 22, 1991 | Masaki |
4991054 | February 5, 1991 | Walters |
5001558 | March 19, 1991 | Burley et al. |
5003288 | March 26, 1991 | Wilhelm |
5012082 | April 30, 1991 | Watanabe |
5016977 | May 21, 1991 | Baude et al. |
5027001 | June 25, 1991 | Torbert |
5027200 | June 25, 1991 | Petrossian et al. |
5044706 | September 3, 1991 | Chen |
5055668 | October 8, 1991 | French |
5059877 | October 22, 1991 | Teder |
5064274 | November 12, 1991 | Alten |
5072154 | December 10, 1991 | Chen |
5086253 | February 4, 1992 | Lawler |
5096287 | March 17, 1992 | Kakinami et al. |
5097362 | March 17, 1992 | Lynas |
5121200 | June 9, 1992 | Choi |
5124549 | June 23, 1992 | Michaels et al. |
5130709 | July 14, 1992 | Toyama et al. |
5168378 | December 1, 1992 | Black |
5170374 | December 8, 1992 | Shimohigashi et al. |
5172235 | December 15, 1992 | Wilm et al. |
5177685 | January 5, 1993 | Davis et al. |
5182502 | January 26, 1993 | Slotkowski et al. |
5184956 | February 9, 1993 | Langlais et al. |
5189561 | February 23, 1993 | Hong |
5193000 | March 9, 1993 | Lipton et al. |
5193029 | March 9, 1993 | Schofield |
5204778 | April 20, 1993 | Bechtel |
5208701 | May 4, 1993 | Maeda |
5245422 | September 14, 1993 | Borcherts et al. |
5255442 | October 26, 1993 | Schierbeek et al. |
5276389 | January 4, 1994 | Levers |
5285060 | February 8, 1994 | Larson et al. |
5289182 | February 22, 1994 | Brillard et al. |
5289321 | February 22, 1994 | Secor |
5305012 | April 19, 1994 | Faris |
5307136 | April 26, 1994 | Saneyoshi |
5309137 | May 3, 1994 | Kajiwara |
5313072 | May 17, 1994 | Vachss |
5325096 | June 28, 1994 | Pakett |
5325386 | June 28, 1994 | Jewell et al. |
5329206 | July 12, 1994 | Slotkowski et al. |
5331312 | July 19, 1994 | Kudoh |
5336980 | August 9, 1994 | Levers |
5341437 | August 23, 1994 | Nakayama |
5351044 | September 27, 1994 | Mathur et al. |
5355118 | October 11, 1994 | Fukuhara |
5374852 | December 20, 1994 | Parkes |
5386285 | January 31, 1995 | Asayama |
5394333 | February 28, 1995 | Kao |
5406395 | April 11, 1995 | Wilson et al. |
5410346 | April 25, 1995 | Saneyoshi et al. |
5414257 | May 9, 1995 | Stanton |
5414461 | May 9, 1995 | Kishi et al. |
5416313 | May 16, 1995 | Larson et al. |
5416318 | May 16, 1995 | Hegyi |
5416478 | May 16, 1995 | Morinaga |
5424952 | June 13, 1995 | Asayama |
5426294 | June 20, 1995 | Kobayashi et al. |
5430431 | July 4, 1995 | Nelson |
5434407 | July 18, 1995 | Bauer et al. |
5440428 | August 8, 1995 | Hegg et al. |
5444478 | August 22, 1995 | Lelong et al. |
5451822 | September 19, 1995 | Bechtel et al. |
5457493 | October 10, 1995 | Leddy et al. |
5461357 | October 24, 1995 | Yoshioka et al. |
5461361 | October 24, 1995 | Moore |
5469298 | November 21, 1995 | Suman et al. |
5471515 | November 28, 1995 | Fossum et al. |
5475494 | December 12, 1995 | Nishida et al. |
5497306 | March 5, 1996 | Pastrick |
5498866 | March 12, 1996 | Bendicks et al. |
5500766 | March 19, 1996 | Stonecypher |
5510983 | April 23, 1996 | Iino |
5515448 | May 7, 1996 | Nishitani |
5521633 | May 28, 1996 | Nakajima et al. |
5528698 | June 18, 1996 | Kamei et al. |
5529138 | June 25, 1996 | Shaw et al. |
5530240 | June 25, 1996 | Larson et al. |
5530420 | June 25, 1996 | Tsuchiya et al. |
5535314 | July 9, 1996 | Alves et al. |
5537003 | July 16, 1996 | Bechtel et al. |
5539397 | July 23, 1996 | Asanuma et al. |
5541590 | July 30, 1996 | Nishio |
5550677 | August 27, 1996 | Schofield et al. |
5555555 | September 10, 1996 | Sato et al. |
5568027 | October 22, 1996 | Teder |
5574443 | November 12, 1996 | Hsieh |
5581464 | December 3, 1996 | Woll et al. |
5594222 | January 14, 1997 | Caldwell |
5610756 | March 11, 1997 | Lynam et al. |
5614788 | March 25, 1997 | Mullins |
5619370 | April 8, 1997 | Guinosso |
5632092 | May 27, 1997 | Blank et al. |
5634709 | June 3, 1997 | Iwama |
5642299 | June 24, 1997 | Hardin et al. |
5648835 | July 15, 1997 | Uzawa |
5650944 | July 22, 1997 | Kise |
5660454 | August 26, 1997 | Mori et al. |
5661303 | August 26, 1997 | Teder |
5666028 | September 9, 1997 | Bechtel et al. |
5670935 | September 23, 1997 | Schofield et al. |
5677851 | October 14, 1997 | Kingdon et al. |
5699044 | December 16, 1997 | Van Lente et al. |
5724316 | March 3, 1998 | Brunts |
5732379 | March 24, 1998 | Eckert et al. |
5737226 | April 7, 1998 | Olson et al. |
5760828 | June 2, 1998 | Cortes |
5760931 | June 2, 1998 | Saburi et al. |
5761094 | June 2, 1998 | Olson et al. |
5765116 | June 9, 1998 | Wilson-Jones et al. |
5765118 | June 9, 1998 | Fukatani |
5781437 | July 14, 1998 | Wiemer et al. |
5786772 | July 28, 1998 | Schofield et al. |
5790403 | August 4, 1998 | Nakayama |
5790973 | August 4, 1998 | Blaker et al. |
5793308 | August 11, 1998 | Rosinski et al. |
5793420 | August 11, 1998 | Schmidt |
5796094 | August 18, 1998 | Schofield et al. |
5835255 | November 10, 1998 | Miles |
5837994 | November 17, 1998 | Stam et al. |
5844505 | December 1, 1998 | Van Ryzin |
5844682 | December 1, 1998 | Kiyomoto et al. |
5845000 | December 1, 1998 | Breed et al. |
5848802 | December 15, 1998 | Breed et al. |
5850176 | December 15, 1998 | Kinoshita et al. |
5850254 | December 15, 1998 | Takano et al. |
5867591 | February 2, 1999 | Onda |
5877707 | March 2, 1999 | Kowalick |
5877897 | March 2, 1999 | Schofield et al. |
5878357 | March 2, 1999 | Sivashankar et al. |
5878370 | March 2, 1999 | Olson |
5883739 | March 16, 1999 | Ashihara et al. |
5884212 | March 16, 1999 | Lion |
5890021 | March 30, 1999 | Onoda |
5896085 | April 20, 1999 | Mori et al. |
5899956 | May 4, 1999 | Chan |
5915800 | June 29, 1999 | Hiwatashi et al. |
5923027 | July 13, 1999 | Stam et al. |
5924212 | July 20, 1999 | Domanski |
5959555 | September 28, 1999 | Furuta |
5963247 | October 5, 1999 | Banitt |
5986796 | November 16, 1999 | Miles |
5990469 | November 23, 1999 | Bechtel et al. |
5990649 | November 23, 1999 | Nagao et al. |
6020704 | February 1, 2000 | Buschur |
6049171 | April 11, 2000 | Stam et al. |
6052124 | April 18, 2000 | Stein et al. |
6066933 | May 23, 2000 | Ponziana |
6084519 | July 4, 2000 | Coulling et al. |
6097024 | August 1, 2000 | Stam et al. |
6100799 | August 8, 2000 | Fenk |
6144022 | November 7, 2000 | Tenenbaum et al. |
6175300 | January 16, 2001 | Kendrick |
6178034 | January 23, 2001 | Allemand et al. |
6201642 | March 13, 2001 | Bos et al. |
6223114 | April 24, 2001 | Boros et al. |
6227689 | May 8, 2001 | Miller |
6250148 | June 26, 2001 | Lynam |
6266082 | July 24, 2001 | Yonezawa et al. |
6266442 | July 24, 2001 | Laumeyer et al. |
6285393 | September 4, 2001 | Shimoura et al. |
6294989 | September 25, 2001 | Schofield et al. |
6297781 | October 2, 2001 | Turnbull et al. |
6310611 | October 30, 2001 | Caldwell |
6313454 | November 6, 2001 | Bos et al. |
6317057 | November 13, 2001 | Lee |
6320282 | November 20, 2001 | Caldwell |
6333759 | December 25, 2001 | Mazzilli |
6341523 | January 29, 2002 | Lynam |
6353392 | March 5, 2002 | Schofield et al. |
6370329 | April 9, 2002 | Teuchert |
6392315 | May 21, 2002 | Jones et al. |
6396397 | May 28, 2002 | Bos et al. |
6411204 | June 25, 2002 | Bloomfield et al. |
6420975 | July 16, 2002 | DeLine et al. |
6424273 | July 23, 2002 | Gutta et al. |
6430303 | August 6, 2002 | Naoi et al. |
6442465 | August 27, 2002 | Breed et al. |
6477464 | November 5, 2002 | McCarthy et al. |
6497503 | December 24, 2002 | Dassanayake et al. |
6498620 | December 24, 2002 | Schofield et al. |
6516664 | February 11, 2003 | Lynam |
6534884 | March 18, 2003 | Marcus et al. |
6539306 | March 25, 2003 | Turnbull |
6547133 | April 15, 2003 | DeVries, Jr. et al. |
6553130 | April 22, 2003 | Lemelson et al. |
6574033 | June 3, 2003 | Chui et al. |
6589625 | July 8, 2003 | Kothari et al. |
6594583 | July 15, 2003 | Ogura et al. |
6611610 | August 26, 2003 | Stam et al. |
6636258 | October 21, 2003 | Strumolo |
6650455 | November 18, 2003 | Miles |
6672731 | January 6, 2004 | Schnell et al. |
6674562 | January 6, 2004 | Miles |
6678614 | January 13, 2004 | McCarthy et al. |
6680792 | January 20, 2004 | Miles |
6690268 | February 10, 2004 | Schofield et al. |
6700605 | March 2, 2004 | Toyoda et al. |
6704621 | March 9, 2004 | Stein et al. |
6710908 | March 23, 2004 | Miles et al. |
6711474 | March 23, 2004 | Treyz et al. |
6714331 | March 30, 2004 | Lewis et al. |
6717610 | April 6, 2004 | Bos et al. |
6735506 | May 11, 2004 | Breed et al. |
6741377 | May 25, 2004 | Miles |
6744353 | June 1, 2004 | Sjönell |
6757109 | June 29, 2004 | Bos |
6762867 | July 13, 2004 | Lippert et al. |
6794119 | September 21, 2004 | Miles |
6795221 | September 21, 2004 | Urey |
6806452 | October 19, 2004 | Bos et al. |
6819231 | November 16, 2004 | Berberich et al. |
6823241 | November 23, 2004 | Shirato et al. |
6824281 | November 30, 2004 | Schofield et al. |
6850156 | February 1, 2005 | Bloomfield et al. |
6882287 | April 19, 2005 | Schofield |
6889161 | May 3, 2005 | Winner et al. |
6909753 | June 21, 2005 | Meehan et al. |
6946978 | September 20, 2005 | Schofield |
6968736 | November 29, 2005 | Lynam |
6975775 | December 13, 2005 | Rykowski et al. |
6989736 | January 24, 2006 | Berberich et al. |
7004606 | February 28, 2006 | Schofield |
7005974 | February 28, 2006 | McMahon et al. |
7038577 | May 2, 2006 | Pawlicki et al. |
7062300 | June 13, 2006 | Kim |
7065432 | June 20, 2006 | Moisel et al. |
7079017 | July 18, 2006 | Lang et al. |
7085637 | August 1, 2006 | Breed et al. |
7092548 | August 15, 2006 | Laumeyer et al. |
7111968 | September 26, 2006 | Bauer et al. |
7113867 | September 26, 2006 | Stein et al. |
7116246 | October 3, 2006 | Winter et al. |
7123168 | October 17, 2006 | Schofield |
7136753 | November 14, 2006 | Samukawa et al. |
7145519 | December 5, 2006 | Takahashi et al. |
7149613 | December 12, 2006 | Stam et al. |
7151996 | December 19, 2006 | Stein et al. |
7161616 | January 9, 2007 | Okamoto et al. |
7195381 | March 27, 2007 | Lynam et al. |
7202776 | April 10, 2007 | Breed |
7227611 | June 5, 2007 | Hull et al. |
7338177 | March 4, 2008 | Lynam |
7365769 | April 29, 2008 | Mager |
7370983 | May 13, 2008 | De Wind et al. |
7437244 | October 14, 2008 | Okada |
7460951 | December 2, 2008 | Altan |
7480149 | January 20, 2009 | DeWard et al. |
7490007 | February 10, 2009 | Taylor et al. |
7492281 | February 17, 2009 | Lynam et al. |
7502048 | March 10, 2009 | Okamoto et al. |
7526103 | April 28, 2009 | Schofield et al. |
7532109 | May 12, 2009 | Takahama et al. |
7566851 | July 28, 2009 | Stein et al. |
7581859 | September 1, 2009 | Lynam |
7592928 | September 22, 2009 | Chinomi et al. |
7639149 | December 29, 2009 | Katoh |
7681960 | March 23, 2010 | Wanke et al. |
7720580 | May 18, 2010 | Higgins-Luthman |
7724962 | May 25, 2010 | Zhu et al. |
7777611 | August 17, 2010 | Desai |
7786898 | August 31, 2010 | Stein et al. |
7855755 | December 21, 2010 | Weller et al. |
7881496 | February 1, 2011 | Camilleri et al. |
7914187 | March 29, 2011 | Higgins-Luthman et al. |
7949486 | May 24, 2011 | Denny et al. |
7952490 | May 31, 2011 | Fechner et al. |
7965336 | June 21, 2011 | Bingle et al. |
8013780 | September 6, 2011 | Lynam et al. |
8027029 | September 27, 2011 | Lu et al. |
8058977 | November 15, 2011 | Lynam |
8064643 | November 22, 2011 | Stein et al. |
8082101 | December 20, 2011 | Stein et al. |
8164628 | April 24, 2012 | Stein et al. |
8254635 | August 28, 2012 | Stein et al. |
8340866 | December 25, 2012 | Hanzawa et al. |
8378851 | February 19, 2013 | Stein et al. |
8452055 | May 28, 2013 | Stein et al. |
8487991 | July 16, 2013 | Zhang et al. |
8553088 | October 8, 2013 | Stein et al. |
8849495 | September 30, 2014 | Chundrlik, Jr. et al. |
20020015153 | February 7, 2002 | Downs |
20020044065 | April 18, 2002 | Quist et al. |
20020113873 | August 22, 2002 | Williams |
20030137586 | July 24, 2003 | Lewellen |
20030222982 | December 4, 2003 | Hamdan et al. |
20040032321 | February 19, 2004 | McMahon et al. |
20040114381 | June 17, 2004 | Salmeen et al. |
20050078389 | April 14, 2005 | Kulas et al. |
20050134983 | June 23, 2005 | Lynam |
20050165550 | July 28, 2005 | Okada |
20050264891 | December 1, 2005 | Uken et al. |
20060018511 | January 26, 2006 | Stam et al. |
20060018512 | January 26, 2006 | Stam et al. |
20060061008 | March 23, 2006 | Karner et al. |
20060091813 | May 4, 2006 | Stam et al. |
20060103727 | May 18, 2006 | Tseng |
20060164221 | July 27, 2006 | Jensen |
20060250501 | November 9, 2006 | Wildmann et al. |
20060290479 | December 28, 2006 | Akatsuka et al. |
20070104476 | May 10, 2007 | Yasutomi et al. |
20070274566 | November 29, 2007 | Fujimoto |
20090093938 | April 9, 2009 | Isaji et al. |
20090113509 | April 30, 2009 | Tseng et al. |
20090177347 | July 9, 2009 | Breuer et al. |
20090189783 | July 30, 2009 | Koitabashi |
20090243824 | October 1, 2009 | Peterson et al. |
20090244361 | October 1, 2009 | Gebauer et al. |
20090252377 | October 8, 2009 | Akita |
20090265069 | October 22, 2009 | Desbrunes |
20090279740 | November 12, 2009 | Yamazaki |
20090295181 | December 3, 2009 | Lawlor et al. |
20100020170 | January 28, 2010 | Higgins-Luthman et al. |
20100097469 | April 22, 2010 | Blank et al. |
20100228437 | September 9, 2010 | Hanzawa et al. |
20110115912 | May 19, 2011 | Kuehnle et al. |
20110216201 | September 8, 2011 | McAndrew et al. |
20120044066 | February 23, 2012 | Mauderer et al. |
20120062743 | March 15, 2012 | Lynam et al. |
20120218412 | August 30, 2012 | Dellantoni et al. |
20120245817 | September 27, 2012 | Cooprider et al. |
20120262340 | October 18, 2012 | Hassan et al. |
20120303222 | November 29, 2012 | Cooprider et al. |
20130124052 | May 16, 2013 | Hahne |
20130129150 | May 23, 2013 | Saito |
20130131918 | May 23, 2013 | Hahne |
20140067206 | March 6, 2014 | Pflug |
20140156157 | June 5, 2014 | Johnson et al. |
20140222280 | August 7, 2014 | Salomonsson |
20140313339 | October 23, 2014 | Diessner et al. |
20140350834 | November 27, 2014 | Turk |
20140379233 | December 25, 2014 | Chundrlik, Jr. et al. |
2007129560 | May 2007 | JP |
2010020476 | January 2010 | JP |
2010038907 | February 2010 | JP |
WO2010119329 | October 2010 | WO |
WO2012145818 | January 2012 | WO |
- International Search Report dated Mar. 23, 2012 for corresponding PCT Application No. PCT/CA2012/00057.
Type: Grant
Filed: Jan 20, 2012
Date of Patent: Jan 17, 2017
Patent Publication Number: 20140169627
Assignee: MAGNA ELECTRONICS INC. (Auburn Hills)
Inventor: Nikhil Gupta (Brampton)
Primary Examiner: Vikkram Bali
Assistant Examiner: Tracy Mangialaschi
Application Number: 14/113,415
International Classification: G06K 9/00 (20060101); G06T 7/20 (20060101);