Control of host device using three-dimensional position and velocity
A system and method of controlling operation of a host device in real-time, the host device operatively connected to an optical device and a radar device. The optical device is configured to obtain visual data of at least one object. The object is located at an incline, relative to the host device, the incline being characterized by an elevation angle (ϕ) and an azimuth angle (θ). The radar device is configured to obtain radar data, including a radial distance (r) of the object from the host device, the azimuth angle (θ), and a range rate (dr/dt). The controller is programmed to determine a time-to-contact for the host device and the object based at least partially on a 3-D position and 3-D velocity vector. The operation of the host device is controlled based at least partially on the time-to-contact.
Latest General Motors Patents:
- AUDIO SIGNAL TRANSMISSION WITH DYNAMIC SOURCE AND TARGET POSITIONS IN A VEHICLE
- HARMONIC CURRENT COMMAND WITH FOUR DEGREES OF FREEDOM FOR ELECTRIC MOTOR
- DC-DC POWER CONVERTER PRE-CHARGE SYSTEM
- COLUMNAR SILICON ANODE HAVING A CARBONACEOUS NETWORK AND METHODS OF FORMING THE SAME
- ARTICULATING ROOF ASSEMBLIES FOR ELECTRICAL GENERATORS AND VEHICLE CHARGING STATIONS
The present disclosure relates to a system and method of controlling operation of a host device in real-time, using three-dimensional position and velocity information of at least one neighboring object. Mobile devices, including but not limited to motor vehicles, may be surrounded by other moving and non-moving objects. If the surrounding objects are positioned at an incline relative to the host device, it may be challenging to ascertain their respective trajectories.
SUMMARYA system and method of controlling operation of a host device in real-time includes an optical device operatively connected to the host device. The optical device is configured to obtain visual data, including one or more images of at least one object. The object is located at an incline, relative to the host device, the incline being characterized by an elevation angle (ϕ) and an azimuth angle (θ). A radar device is operatively connected to the host device and configured to obtain radar data. The radar data includes a radial distance (r) of the object from the host device, the azimuth angle (θ), and a range rate (dr/dt).
A controller is operatively connected to the host device and includes a processor and tangible, non-transitory memory on which is recorded instructions. Execution of the instructions by the processor causes the controller to obtain respective bounding boxes around the object in the visual data. The controller is programmed to determine an optical flow of the respective bounding boxes, including determining a change rate of the elevation angle (dϕ/dt) and a change rate of the azimuth angle (dθ/dt).
The controller is programmed to determine a time-to-contact for the host device and the object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (vx, vy, vz). The operation of the host device is controlled based at least partially on the time-to-contact.
The host device may include an automatic braking module configured to decelerate the host device. Controlling operation of the host device may include sending a signal to a user of the host device, if the time-to-contact is less than a first predefined threshold but greater than a second predefined threshold. Controlling operation of the host device may include activating the automatic braking module, if the time-to-contact is less than the second predefined threshold.
The controller may be programmed to communicate with a remote module such that the remote module is not physically connected to the controller. The controller may be programmed to transfer a location coordinate of the host device to the remote module and receive a location assessment factor (LAF) from the remote module, based at least partially on the location coordinate. In one embodiment, the controller is programmed to determine the time-to-contact only if the location assessment factor (LAF) is at or above a threshold value.
The system is configured to obtain three-dimensional position and velocity information for the object, by conjoining the radar data and the visual data. Because the position and velocity information is obtained in three dimensions, accuracy of the time-to-contact between the host device and the respective objects is significantly improved.
The controller may be programmed to match the respective bounding boxes in the visual data with the radar data. The controller may include an association module configured to perform the matching of the respective bounding boxes with the radar data. The controller is programmed to determine a 3-D position matrix (r, θ, ϕ) and a 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt), by combining the visual data and the radar data. The controller may include a Kalman filter module configured to perform the combining of the visual data and the radar data. A 3-D position vector (X, Y, Z) and a 3-D velocity vector (vx, vy, vz) are obtained from the 3-D position matrix (r, θ, ϕ) and the 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt), respectively.
The respective bounding boxes include a first bounding box B1 characterized by an initial position (u,v). The optical flow may be characterized by a first displacement change (Δu) of the respective bounding boxes in row coordinates and a second displacement change (Δv) of the respective bounding boxes in column coordinates, both over a predefined unit of time (Δt). The 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt) include respective components determined as a function of the optical flow, the initial position (u,v), an optical focal length (f) and the predefined unit of time (Δt).
The above features and advantages and other features and advantages of the present disclosure are readily apparent from the following detailed description of the best modes for carrying out the disclosure when taken in connection with the accompanying drawings.
Referring to the drawings, wherein like reference numbers refer to like components,
Referring to
Referring to
Referring to
Referring to
Referring now to
In block 102A of
In block 102B of
In block 104, the controller C is programmed to obtain respective bounding boxes around the object 20 in each of the images in the visual data. Referring to
In block 104 of
Block 104 includes determining the change rate of the elevation angle (dϕ/dt) and the change rate of the azimuth angle (dθ/dt). The elevation angle (ϕ), azimuth angle (θ), the change rate of the elevation angle (dϕ/dt) and the change rate of the azimuth angle (dθ/dt) may be determined as a function of the optical flow, the initial position (u,v), an optical focal length (f) .e., the projection of the radial distance (r) on the X-Y plane in the first coordinate system 24 and the predefined unit of time (Δt) such that:
(dϕ/dt)=Δu/(f*Δt);
(dθ/dt)=Δv/(f*Δt);
ϕ=inverse tangent(u/f); and
θ=inverse tangent(v/f).
In block 106 of
In block 108 of
In block 110 of
x=r sin ϕ cos θ;y=r sin θ sin ϕ;z=r cos ϕ;
vx=[(dr/dt)sin ϕ cos θ+r(dθ/dt)cos θ cos ϕ−r(dϕ/dt)sin θ sin ϕ];
vy=[(dr/dt)sin θ sin ϕ+r(dθ/dt)cos θ sin ϕ+r(dϕ/dt)sin θ cos ϕ];
vz=[(dr/dt)cos ϕ−r(dθ/dt)sin θ].
From block 110, the method 100 may proceed to block 111 (or block 114 directly, as shown by line 112). In block 111, the controller C is programmed to transmit a location coordinate (i.e., geophysical coordinates) of the host device 12 to the remote module 30 (see
From block 111, the method 100 proceeds to block 113, where the controller C is programmed to determine if the location assessment factor (LAF) is at or above a threshold value (e.g. 0.5). In one embodiment, the controller C is programmed to proceed to block 114 and determine the time-to-contact as long as the location assessment factor (LAF) is at or above the threshold value.
As noted above, the method 100 may proceed to block 114 directly from block 110 or from block 113. In block 114 of
t=√{square root over ((X/vx)2+(Y/vy)2+(Z/vz)2)}
In block 116 of
The controller C (and execution of the method 100) improves the functioning of the host device 12 by precisely obtaining time-to-contact between the host device 12 and the respective ones of the plurality of objects 20 and controlling operation of the host device 12 based at least partially on the time-to-contact. The controller C of
The controller C includes a computer-readable medium (also referred to as a processor-readable medium), including a non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random access memory (DRAM), which may constitute a main memory. Such instructions may be transmitted by one or more transmission media, including coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to a processor of a computer. Some forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, other magnetic media, a CD-ROM, DVD, other optical media, punch cards, paper tape, other physical media with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, other memory chips or cartridges, or other media from which a computer can read.
Look-up tables, databases, data repositories or other data stores described herein may include various kinds of mechanisms for storing, accessing, and retrieving various kinds of data, including a hierarchical database, a set of files in a file system, an application database in a proprietary format, a relational database management system (RDBMS), etc. Each such data store may be included within a computing device employing a computer operating system such as one of those mentioned above, and may be accessed via a network in one or more of a variety of manners. A file system may be accessible from a computer operating system, and may include files stored in various formats. An RDBMS may employ the Structured Query Language (SQL) in addition to a language for creating, storing, editing, and executing stored procedures, such as the PL/SQL language mentioned above.
The detailed description and the drawings or FIGS. are supportive and descriptive of the disclosure, but the scope of the disclosure is defined solely by the claims. While some of the best modes and other embodiments for carrying out the claimed disclosure have been described in detail, various alternative designs and embodiments exist for practicing the disclosure defined in the appended claims. Furthermore, the embodiments shown in the drawings or the characteristics of various embodiments mentioned in the present description are not necessarily to be understood as embodiments independent of each other. Rather, it is possible that each of the characteristics described in one of the examples of an embodiment can be combined with one or a plurality of other desired characteristics from other embodiments, resulting in other embodiments not described in words or by reference to the drawings. Accordingly, such other embodiments fall within the framework of the scope of the appended claims.
Claims
1. A system of controlling operation of a host device in real-time, the system comprising:
- an optical device operatively connected to the host device and configured to obtain visual data of at least one object, the at least one object being located at an incline relative to the host device, the incline being characterized by an elevation angle (ϕ) and an azimuth angle (θ);
- a radar device operatively connected to the host device and configured to obtain radar data, including the azimuth angle (θ), a radial distance (r) of the at least one object from the host device and a change rate of the range (dr/dt);
- a controller operatively connected to the host device and including a processor and tangible, non-transitory memory on which is recorded instructions, execution of the instructions by the processor causing the controller to: obtain respective bounding boxes around the at least one object in the visual data, the respective bounding boxes including a first bounding box characterized by an initial position (u,v); match the respective bounding boxes with the radar data; determine an optical flow of the respective bounding boxes, including determining a change rate of the elevation angle (dϕ/dt) and a change rate of the azimuth angle (dθ/dt), the optical flow being characterized by a first displacement change (Δu) of the first bounding box in row coordinates and a second displacement change (Δv) of the first bounding box in column coordinates, both over a predefined unit of time (Δt); determine a 3-D position matrix (r, θ, ϕ) and a 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt) based on the visual data and the radar data, including determining respective components of the 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt) as a function of the optical flow, the initial position (u,v), an optical focal length (f) and the predefined unit of time (Δt) such that: ϕ=inverse tangent (u/f), θ=inverse tangent (v/f), (dθ/dt)=Δv/(f*Δt) and (dϕ/dt) =Δu/(f*Δt); and control operation of the host device based at least partially on the 3-D position matrix (r, θ, ϕ) and the 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt).
2. The system of claim 1, wherein the controller is configured to:
- obtain a 3-D position vector (X, Y, Z) and a 3-D velocity vector (vx, vy, vz) from the 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt); and
- determine a time-to-contact for the host device and the at least one object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (vx, vy, vz).
3. The system of claim 2, wherein the controller is configured to:
- communicate with a remote module such that the remote module is not physically connected to the controller;
- transfer a location coordinate of the host device to the remote module;
- receive a location assessment factor (LAF) from the remote module, based at least partially on the location coordinate; and
- determine the time-to-contact only if the location assessment factor (LAF) is at or above a threshold value.
4. The system of claim 2, wherein the controlling operation of the host device includes:
- sending a signal to a user of the host device, if the time-to-contact is less than a first predefined threshold but greater than a second predefined threshold; and
- wherein the first predefined threshold is greater than the second predefined threshold.
5. The system of claim 4, wherein:
- the host device includes an automatic braking module configured to decelerate the host device; and
- controlling operation of the host device includes activating the automatic braking module, if the time-to-contact is less than the second predefined threshold.
6. The system of claim 2, wherein:
- the controller includes an association module configured to perform the matching of the respective bounding boxes with the radar data.
7. The system of claim 2, wherein:
- the controller includes a Kalman filter module configured to perform the combining of the visual data and the radar data.
8. The system of claim 2, wherein the time-to-contact (t) is defined as:
- t=√{square root over ((X/vx)2+(Y/vy)2+(Z/vz)2)}.
9. The system of claim 2, wherein respective components of the 3-D position vector (X, Y, Z) and the 3-D velocity vector (vx, vy, vz) are defined as:
- X=r sin ϕ cos θ;
- Y=r sin θ sin ϕ;
- Z=r cos ϕ;
- vx=[(dr/dt)sin ϕ cos θ+r(dθ/dt)cos θ cos ϕ−r(dϕ/dt)sin θ sin ϕ];
- vy=[(dr/dt)sin θ sin ϕ+r(dθ/dt)cos θ sin ϕ+r(dϕ/dt)sin θ cos ϕ]; and
- vz=[(dr/dt)cos ϕ−r(dθ/dt)sin θ].
10. A method of controlling operation of a host device in real-time, the host device operatively connected to an optical device, a radar device and a controller having a processor and tangible, non-transitory memory, the method comprising:
- obtaining visual data of at least one object via the optical device, the at least one object being located at an incline relative to the host device, the incline being characterized by an elevation angle (ϕ) and an azimuth angle (θ);
- obtaining radar data via the radar device, including the azimuth angle (θ), a radial distance (r) of the at least one object from the host device and a change rate of the range (dr/dt);
- obtaining respective bounding boxes around the at least one object in the visual data, the respective bounding boxes including a first bounding box characterized by an initial position (u,v), via the controller;
- matching the respective bounding boxes with the radar data, via the controller;
- determining an optical flow of the respective bounding boxes, via the controller, including determining a change rate of the elevation angle (dϕ/dt) and a change rate of the azimuth angle (dθ/dt), via the controller, the optical flow being characterized by a first displacement change (Δu) of the first bounding box in row coordinates and a second displacement change (Δv) of the first bounding box in column coordinates, both over a predefined unit of time (Δt);
- determining a 3-D position matrix (r, θ, ϕ) and a 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt) based on the visual data and the radar data, via the controller, including determining respective components of the 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt) as a function of the optical flow, the initial position (u,v), an optical focal length (f) and the predefined unit of time (Δt) such that: ϕ=inverse tangent (u/f), θ=inverse tangent (v/f), (dθ/dt) =Δv/(f*Δt) and (dϕ/dt)=Δu/(f*Δt); and
- controlling operation of the host device based at least partially on the 3-D position matrix (r, θ, ϕ) and the 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt).
11. The method of claim 10, wherein the controller is configured to:
- obtain a 3-D position vector (X, Y, Z) and a 3-D velocity vector (vx, vy, vz) from the 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt); and
- determine a time-to-contact for the host device and the at least one object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (vx, vy, vz).
12. The method of claim 11, wherein the controller is configured to:
- communicate with a remote module such that the remote module is not physically connected to the controller;
- transfer a location coordinate of the host device to the remote module;
- receive a location assessment factor (LAF) from the remote module, based at least partially on the location coordinate; and
- determine the time-to-contact only if the location assessment factor (LAF) is at or above a threshold value.
13. The method of claim 11, wherein the controlling operation of the host device includes:
- sending a signal to a user of the host device, if the time-to-contact is less than a first predefined threshold but greater than a second predefined threshold; and
- wherein the first predefined threshold is greater than the second predefined threshold.
14. The method of claim 13, wherein:
- the host device includes an automatic braking module configured to decelerate the host device; and
- controlling operation of the host device includes activating the automatic braking module, if the time-to-contact is less than the second predefined threshold.
15. The method of claim 11, further comprising:
- the controller includes an association module configured to perform the matching of the respective bounding boxes with the radar data.
16. The method of claim 11, wherein:
- the controller includes a Kalman filter module configured to perform the combining of the visual data and the radar data.
17. The method of claim 11, wherein the time-to-contact (t) is defined as:
- t=√{square root over ((X/vx)2+(Y/vy)2+(Z/vz)2)}.
18. The method of claim 10, further comprising:
- defining the respective components of the 3-D position vector (X, Y, Z) and the 3-D velocity vector (vx, vy, vz) as: X=r sin ϕ cos θ; Y=r sin θ sin ϕ; Z=r cos ϕ; vx=[(dr/dt)sin ϕ cos θ+r(dθ/dt)cos θ cos ϕ−r(dϕ/dt)sin θ sin ϕ]; vy=[(dr/dt)sin θ sin ϕ+r(dθ/dt)cos θ sin ϕ+r(dϕ/dt)sin θ cos ϕ]; and vz=[(dr/dt)cos ϕ−r(dθ/dt)sin θ].
9429650 | August 30, 2016 | Zeng |
9708004 | July 18, 2017 | Weisswange |
20030210807 | November 13, 2003 | Sato |
20100191391 | July 29, 2010 | Zeng |
20120027258 | February 2, 2012 | Uchida |
20120106786 | May 3, 2012 | Shiraishi |
20120140076 | June 7, 2012 | Rosenbaum |
20140037138 | February 6, 2014 | Sato |
20150066348 | March 5, 2015 | Baba |
20150234045 | August 20, 2015 | Rosenblum |
20160054452 | February 25, 2016 | Cosatto |
20170123429 | May 4, 2017 | Levinson |
20180024562 | January 25, 2018 | Bellaiche |
20190034742 | January 31, 2019 | Gokan |
Type: Grant
Filed: Mar 8, 2017
Date of Patent: Oct 15, 2019
Patent Publication Number: 20180259637
Assignee: GM Global Technology Operations LLC. (Detroit, MI)
Inventors: Shuqing Zeng (Sterling Heights, MI), Jay H. Sim (Bloomfield, MI), Mohammad Naserian (Windsor)
Primary Examiner: Marcus E Windrich
Application Number: 15/452,851
International Classification: G01S 13/86 (20060101); G01S 13/93 (20060101); G05D 1/02 (20060101); B60W 10/18 (20120101); B60W 30/09 (20120101); G06K 9/00 (20060101);