CALIBRATION OF CAMERA AND PROJECTOR
Described examples include a system including a projector configured to project a test pattern image, the test pattern image having at least two elements; a camera configured to capture the test pattern image; and a controller coupled to the projector and to the camera. The controller is configured to obtain a first calibration matrix between the projector and the camera for the at least two elements; determine at least two epipolar lines based on the first calibration matrix and the test pattern image; determine a cost function based on the at least two epipolar lines and the at least two elements in the test pattern image as captured by the camera; and determine a second calibration matrix responsive to the cost function, wherein at least one of a camera position of the camera or a projector position of the projector is adjusted responsive to the second calibration matrix.
This application claims the benefit of and priority to co-owned U.S. Provisional Patent Application No. 63/079,498, filed Sep. 17, 2020, entitled “An Efficient Scheme for Dynamic Calibration of a Projector & Camera Pair,” which Application is hereby incorporated herein by reference in its entirety.
SUMMARYIn accordance with an example, a system includes a projector configured to project a test pattern image, the test pattern image including at least two elements. The system also includes a camera configured to capture the test pattern image; and a controller coupled to the projector and to the camera. The controller is configured to obtain a first calibration matrix between the projector and the camera for the at least two elements and determine at least two epipolar lines based on the first calibration matrix and the test pattern image. The controller is also configured to determine a cost function based on the at least two epipolar lines and the at least two elements in the captured test pattern image; and determine a second calibration matrix responsive to the cost function, where the second calibration matrix indicates an adjustment of at least one of a camera position of the camera or a projector position of the projector.
In the drawings, corresponding numerals and symbols generally refer to corresponding parts unless otherwise indicated. The drawings are not necessarily drawn to scale.
In this description, the term “coupled” may include connections made with intervening elements, and additional elements and various connections may exist between any elements that are “coupled.” Elements referred to herein as “optically coupled” are elements that include a connection between the elements that involves transmission of light. Also, as used herein, the terms “on” and “over” may include layers or other elements where intervening or additional elements are between an element and the element that it is “on” or “over.”
Projector and camera pairs are used in many applications to provide information and control in many environments. For example, in industrial applications, a projector projects a pattern onto an object. The camera then captures an image of the object with the projected pattern. Analysis of the reflected pattern can then precisely locate the object in multiple dimensions and determine the nature of the object. Automotive applications, such as projector headlights, are another area where camera and projector pairs are useful. Among the uses of projector headlights is providing bright illumination that avoids blinding a driver in a vehicle in front of the projector headlights. A camera determines the shape and position of the vehicle ahead of the projector headlights and then the headlight controller modifies the projected light to avoid the rear window and rear-view mirrors of the vehicle ahead of the projector headlights. However, the pattern of the headlights is aligned to the vehicle in front of the projector headlights. This means that the camera and the projector headlights are aligned to provide the appropriate projected pattern. The initial alignment of the camera and projector is made at the factory and/or at a dealership. However, the motion of the vehicle, bump roads, weather and other factors may cause misalignment while the vehicle is in use. Therefore, it is important to provide a mechanism to align the camera and projector headlights during or after the vehicle being used.
In an example, controller 216 analyzes an image captured by camera 205 to determine what object is ahead of first vehicle 102 (
For each object, a line perpendicular to the epipolar line is generated between the object and the epipolar line. For example, a line in any two-dimensional system (such as camera plane 304 (
ax+by+c=0 (1)
where a is the slope relative to the x axis, b is the slope relative to the y axis and c is a shift from the origin. The shortest distance from the epipolar line to its object is determined by Equation 2:
In
From this information, a cost function is determined using Equation 3:
where n is the number of objects identified. Equation 3 produces a normalized summation of the distances from the objects and their respective epipolar line. The next step is to determine an adjustment to the calibration matrix/pose between the projector plane 302 (
where the left side is a composite rotation matrix and t is translation matrix. The composite rotation matrix is derived from the Euler angles as described hereinbelow. (See Slabaugh, “Computing Euler angles from a rotation matrix,” https://www.gregslabaugh.net/publications/euler.pdf (1999), Wolfram, “Euler Angles, https://mathworld.wolfram.com/EulerAngles.html (last updated Jan. 3, 2021), or Wikipedia, “Rotation Matrix,” https://en.wikipedia.org/wiki/Rotation_matrix, (Jan. 6, 2021), which are hereby incorporated herein by reference in their entirety.). Because the distance from the camera 205 (
R=R(θy)R(θx)R(θz) (5)
where θy, θx, and θz are the Euler angles of the relative positions of the camera and projector. In an example, θy, θx, and θz are rotational parameters. The three rotation matrixes have the format:
(See Slabaugh, Wolfram and Wikipedia cited above.) Solving for the Euler angles yields Equations 9-11:
θy=tan−1(−r31/√{square root over (r112+r212))} (9)
θx=tan−1(r21/r11) (10)
θz=tan−1(r32/r33). (11)
These Euler angles are the result of resolving the rotation matrix R of Matrix 4.
To determine a correction factor to correct a misalignment like that shown in
Step 710 is shown in detail in
Returning to
In an example, controller 816 analyzes an image captured by camera 805 of the image projected on screen 804 using image processor 820 under the control of controlling processor 822. When adjusting the zoom or focal length of projector 803 manually, the focal length can become maladjusted. This shows as a shift of elements on the image. A projector, such as projector 803 shows an image such as image 402 on screen 804. Controller 816 determines epipolar lines for each captured object in the image. For example, first epipolar line 536 (
Modifications are possible in the described examples, and other examples are possible, within the scope of the claims.
Claims
1. A system comprising:
- a projector configured to project a test pattern image, the test pattern image comprising at least two elements;
- a camera configured to capture the test pattern image; and
- a controller coupled to the projector and to the camera, the controller configured to: obtain a first calibration matrix between the projector and the camera for the at least two elements; determine at least two epipolar lines based on the first calibration matrix and the test pattern image; determine a cost function based on the at least two epipolar lines and the at least two elements in the captured test pattern image; and determine a second calibration matrix responsive to the cost function, wherein the second calibration matrix indicates an adjustment of at least one of a camera position of the camera or a projector position of the projector.
2. The system of claim 1, wherein determining the cost function comprises determining a first distance between a first epipolar line of the at least two epipolar lines and a first element of the at least two elements and determining a second distance between a second epipolar line of the at least two epipolar lines and a second element of the at least two elements.
3. The system of claim 1, wherein a second cost function of the second calibration matrix is lower than a first cost function of the first calibration matrix.
4. The system of claim 1, further comprising:
- an adjuster coupled to the projector, the adjuster configured to adjust a position of the projector responsive to the second calibration matrix.
5. The system of claim 1, further comprising:
- an adjuster coupled to the camera, the adjuster configured to adjust a position of the camera responsive to the second calibration matrix.
6. The system of claim 1, wherein the cost function is a normalized summation of distances from the at least two elements and corresponding epipolar lines of the at least two epipolar lines.
7. The system of claim 1, wherein the projector is a projector headlight.
8. The system of claim 1, wherein the second calibration matrix corresponds to a lowest cost of the cost function.
9. The system of claim 8, wherein the lowest cost is determined by varying rotational parameters of the first calibration matrix over at least two iterations, determining a cost of the cost function for the at least two iterations, and selecting the iteration having the lowest cost.
10. A vehicle comprising:
- a projector headlight configured to project a test pattern image, the test pattern image comprising at least two elements;
- a camera configured to capture the test pattern image; and
- a controller coupled to the projector headlight and to the camera, the controller configured to: obtain a first calibration matrix between the projector headlight and the camera for the at least two elements; determine at least two epipolar lines based on the first calibration matrix and the test pattern image; determine a cost function based on the at least two epipolar lines and the at least two elements in the captured test pattern image; and determine a second calibration matrix responsive to the cost function, wherein the second calibration matrix indicates an adjustment of at least one of a camera position of the camera or a projector headlight position of the projector headlight.
11. The vehicle of claim 10, wherein determining the cost function comprises determining a first distance between a first epipolar line of the at least two epipolar lines and a first element of the at least two elements and determining a second distance between a second epipolar line of the at least two epipolar lines and a second element of the at least two elements.
12. The vehicle of claim 10, wherein the cost function is a first cost function and a second cost function of the second calibration matrix is lower than the first cost function of the first calibration matrix.
13. The vehicle of claim 10, wherein the cost function is a normalized summation of distances from the at least two elements and corresponding epipolar lines of the at least two epipolar lines.
14. The vehicle of claim 10, wherein the second calibration matrix corresponds to a lowest cost of the cost function.
15. The vehicle of claim 14, wherein the lowest cost is determined by varying rotational parameters of the first calibration matrix over at least two iterations, determining a cost of the cost function for the at least two iterations and selecting the iteration having the lowest cost.
16. The vehicle of claim 10, further comprising:
- an adjuster coupled to the projector headlight, the adjuster configured to adjust a position of the projector headlight responsive to the second calibration matrix.
17. The vehicle of claim 10, further comprising:
- an adjuster coupled to the camera, the adjuster configured to adjust a position of the camera responsive to the second calibration matrix.
18. A controller comprising:
- a processor, the processor configured to: instruct a projector to project a projected image having patterns; receive a captured image of the projected image from a camera; detect the patterns in the captured image; extract independent rotation variables from a calibration matrix based on a relative position of the projector and the camera; perform an iterative search varying the independent rotation variables, each iteration comprising: constructing a rotation matrix based on the independent rotation variables; calculating an Essential matrix and epipolar lines based on the rotation matrix; and calculating a cost function; select a selected iteration having a low cost function; and initiate adjustment of at least one of the projector or the camera in accordance with the selected iteration.
19. The controller of claim 18, wherein the cost function is a normalized summation of distance from the detected patterns and the epipolar lines.
20. The controller of claim 18, the processor is further configured to determine whether the low cost is below a predetermined threshold.
Type: Application
Filed: Feb 11, 2021
Publication Date: Mar 17, 2022
Inventors: Shashank Dabral (Allen, TX), Jaime Rene De La Cruz Vasquez (Carrollton, TX), Jeffrey Matthew Kempf (Dallas, TX)
Application Number: 17/174,056