USER INTERFACE CONTROLLED BY ENVIRONMENTAL CUES
A method comprises receiving a signal containing an environmental cue; correlating the environmental cue to a user interface control element; and generating a user interface control command based on the environmental cue, A device may be controlled by receiving a visual image, processing the image to identify a visual cue, correlating the visual cue with a user interface control, and controlling the device using the user interface control. The visual image may be received from a camera that is part of an automobile, and the user interface control may include a control for a radio to tune to a radio station; a speed control for controlling a speed of the automobile corresponding to a speed sign visual cue; or a navigation system control for controlling a navigation system based on traffic signs. Other variations include an apparatus and computer-readable media including instructions for performing the methods.
Latest CONVERSANT WIRELESS LICENSING S.A.R.L. Patents:
- System for user-friendly access control setup using a protected setup
- Random access channel response handling with aggregated component carriers
- Method and device for verifying the integrity of platform software of an electronic device
- Setting mode of communication
- Fixed HS-DSCH or E-DCH allocation for VoIP (or HS-DSCH without HS-SCCH/E-DCH without E-DPCCH)
This application is a Continuation of U.S. patent application Ser. No. 15/685,200 filed on Aug. 24, 2017, which is a Continuation of U.S. patent application Ser. No. 14/733,910 filed on Jun. 8, 2015, now issued as U.S. Pat. No. 9,766,792, which is a Continuation of U.S. patent application Ser. No. 13/750,894, filed on Jan. 25, 2013, which is a Continuation of U.S. patent application Ser. No. 11/965,136, filed Dec. 27, 2007, now issued as U.S. Pat. No. 8,370,755 on Feb. 5, 2013, which are hereby incorporated by reference.
BACKGROUNDThe technology relates generally to user interfaces that control devices, such as radios, automobiles, navigation systems, and other types of devices.
User interfaces for controlling equipment, such as a car radio, are well known. Such interfaces may comprise various knobs, switches, and on-screen icons and buttons that control the selection of radio volume, station, and audio enhancements. The user typically makes selections by pressing the buttons, knobs, or icons on a touch-screen display. A user interface for controlling the speed of an automobile may include one or more switches or levers that are pressed to activate cruise control or to accelerate to a higher speed. A user interface for controlling a navigation system may include various icons and on-screen selectors for selecting a destination, re-routing to avoid a traffic condition, or requesting more information about a particular point of interest. A user interface on a mobile device such as a mobile phone may include a touch-screen display allowing numerals to be selected and buttons for initiating and terminating a telephone call.
It would be desirable to permit control of a user interface on the basis of environmental cues, such as visual cues like a traffic sign or information sign.
SUMMARYThis summary is not intended to identify any critical or key elements of the invention, but instead merely presents certain introductory concepts so that the full scope of the invention may be appreciated upon reading the full specification, of which this summary is a part.
One embodiment relates to a method comprising steps of receiving a signal containing an environmental cue; correlating the environmental cue to a user interface control element; and generating a user interface control command based on the environmental cue. An apparatus and computer-readable medium comprising executable instructions suitable for carrying out the method are also included.
One embodiment relates to a method for controlling a device by receiving a visual image, processing the image to identify a visual cue within the image, correlating the visual cue with a user interface control, and controlling a device using the user interface control. The visual image may be received from a camera that is part of an automobile, and the user interface control may include a control for a radio to tune to a radio station corresponding to a radio station visual cue; a speed control for controlling a speed of the automobile corresponding to a speed sign visual cue: or a navigation system control for controlling a navigation system based on traffic signs or other indicia in the image. In some embodiments, an object recognition algorithm identifies one or more visual cues from the image. In other embodiments, a user may select or highlight a visual cue from within the image in order to prompt the user interface. In certain embodiments, visual cues in an image may be matched to those stored in a database, wherein the visual cues are categorized according to country or region, each such country or region associated with different visual cues corresponding to a particular user interface command.
In yet other embodiments, instead of a visual image, an electromagnetic signal such as a radio signal may be received to provide an environmental cue, such as a radio signal that indicates a traffic condition or a radio station advertisement.
Another embodiment relates to an apparatus including a camera configured to receive a visual image, a processor configured to receive the visual image and to identify one or more visual cues in the visual image, and a user interface control configured to generate one or more commands for controlling equipment. In some embodiments, an image processing algorithm processes the image in order to identify one or more visual cues and to correlate such cues with one or more user interface control elements. In certain embodiments, a touch-screen display is used to display the visual image, to receive selection by a user of a visual cue, and to thereafter cause a control signal to be generated. The control signal may be used to control any of various parameters, such as the speed of an automobile in which the apparatus is located; the radio station to which a radio is tuned; or a navigation system.
Yet another embodiment relates to computer-readable media comprising executable instructions that, when executed by a processor or other device, perform one or more steps of various methods including those described above.
Other embodiments and variations will be apparent upon reading the detailed description set forth below, and the invention is not intended to be limited in any way by this brief summary.
Camera 102 may be positioned so as to capture an image of a traffic sign 103, such as a speed limit sign. Images captured by camera 102 are processed in processor 101 and optionally displayed to a user as explained in more detail below. According to certain variations, portions of images captured by camera 102 are matched with one or more visual cues stored in a database in order to control one or more devices, such as radio tuner 107, navigation system 108, speed control system 109, or telephone 114.
A touch screen display 106 is also coupled to processor 101 in some embodiments in order to provide visual feedback, such as the current image captured by camera 102, and to receive user input and selection. A non-touch screen display, such as a conventional LCD or LED display, may be provided instead. User input may also be received from a different input device 115, such as a keypad, keyboard, buttons on a steering wheel, voice input unit, or the like. Processor 101 is also coupled to a cues database 110, which stores visual cue patterns, such as speed limit sign patterns. The cue patterns may be organized by different regions 110a and 110b, such that only a region corresponding to the location in which the automobile is traveling will be searched for visual cue patterns. Thus, for example, speed limit signs in one country may differ in appearance from speed limit signs in another country. Visual cues corresponding to speed limit signs may therefore be searched based on the country or region in which the automobile is traveling.
Optionally a radio receiver 104 may receive signals from a transmitter 105, which may be present to transmit advertisements or traffic-related information as explained in more detail below.
In step 202, user input is received indicating selection of a portion of the image. For example, as shown in
In one variation, corresponding to step 203 of
In step 204, the selected portion of the image may be compared to a database of environmental cues, such as a database of traffic signs, in order to find a match. Any of various image-recognition techniques may be employed, and as indicated above and shown in
In step 205, the recognized traffic sign is matched with a user input control. For example, if the selected portion of the image corresponds to a speed limit sign, a matching user input control element could be a SET SPEED LIMIT command to an automobile cruise control system, as illustrated by element 304 in
In one variation, different speed limit sign patterns are stored in database 110, each corresponding to a different speed limit (e.g., 80 kilometers per hour, 90 kilometers per hour, etc.). In other variations, a generic speed limit sign pattern may be detected and optical character recognition software may be used to decode the numeric speed limit contained in the sign. Of course, traffic sign shapes and configurations may vary from one country to the next, so the illustrated signs are not intended to be limiting.
Steps 202 through 205 collectively correspond to a generic step of correlating an environmental cue to a user interface element.
Finally, in step 206, the user input control is applied to the device, such as the cruise control system of the automobile. The user input control may be transmitted to various devices such as on-board systems over any of various means, such as a control bus, wireless link, or the like.
To the extent that more than one type of user input control matches or is applicable, a menu of options can be presented to the user, allowing for easy selection from among a set of choices.
Instead of a touch-screen display, a conventional LCD or LED display may be used, and user input may be received by way of voice control or buttons (e.g., on the steering wheel) to select options and to navigate around portions of the display.
Another type of environmental cue comprises a sign indicating a radio station that is available in the area in which an automobile is traveling. As shown in
Yet another type of environmental cue comprises a dynamic traffic alert, such as an ACCIDENT AHEAD, TRAFFIC AHEAD, or CONSTRUCTION ZONE sign that is presented to drivers traveling along a road. In one variation of the invention, such dynamic traffic alerts can be recognized by processor 101 and used to generate a DETOUR command to a navigation system 108. Consequently, the driver need not be distracted from driving by navigating through a hierarchy of menus on the touch-screen of the navigation system, but can instead be prompted to immediately cause the navigation system to enter its detour mode.
Yet another type of environmental cue comprises a sign containing information concerning a tourist attraction, such as a museum or park. In accordance with one variation of the invention, upon selecting such a sign in the displayed image, processor 101 can generate a search command in a navigation system 108 in order to cause the navigation system to plan a route to the indicated attraction based on the name of the attraction or a telephone number contained on the sign. If a website address is displayed on a sign, processor 101 may automatically generate a web query to retrieve a webpage corresponding to the website and display it on touch screen display 106 or on another display. Wireless links, such as a wireless telephone network or other type of network, may be accessed by processor 101 according to certain embodiments if needed to access such websites. If a telephone number is included on the sign, the user can be prompted to automatically dial the telephone number by pressing a button or pressing on a displayed message offering to dial the number. This provides an easy one-touch or two-touch way of dialing a telephone number (one touch to select the sign, and a second touch to activate dialing of the telephone number).
As another example, an airport sign along a road may show an airplane symbol (e.g., a black airplane on a white background) and such a symbol can be detected and used in connection with current position information to generate a website address corresponding to the particular airport, which is then provided to phone/web browser 114 (see
In certain variations, the sign itself may include a telephone number or website providing a link to flight information, flight schedules, terminal information, or parking availability. Selection of the sign as a visual cue causes the system to link to the website or call the indicated telephone number for information. In one embodiment, upon displaying a webpage corresponding to the visual cue, the user may provide additional information (e.g., flight number and airline via voice input or numeric keypad or keyboard) to obtain the latest flight or parking status. Additionally, a webpage may show the cheapest or most abundant parking lots, which upon selection by the user can be provided as a destination address to the navigation system. To the extent that information is pre-stored or transferred from a mobile device via short-range radio or infrared means, it can also be provided as an input to the website and to as an input to the navigation system.
Yet another type of environmental cue comprises a traffic routing sign, such as a roundabout (traffic circle) sign. Sometimes the navigation system may not have the latest road patterns or turns. As shown in
In yet other embodiments, instead of visual cues, the dynamic location of a vehicle or user with a mobile device can be used to automatically match user interface controls associated with the location. For example, cues database 110 of
In yet other embodiments, radio signals may be received from a transmitter 105 to a radio receiver 104 (see
As explained above, in certain embodiments different regional cues databases 110a and 110b may be active. As the vehicle travels through different regions or countries, different cues databases may become activated in order to narrow the possible matching patterns that are used to compare with portions of images received from the camera.
In yet another variation, the inventive principles are applied not in an automobile context but in a mobile device such as a mobile telephone; personal digital assistant (PDA), mobile computer, and the like. For example, a pedestrian walking along a road can aim a camera at a road sign, such as a sign advertising a radio station, and cause a radio tuner on the mobile device to be tuned to the advertised frequency. As another example, a user may aim a camera contained within a mobile device at a sign advertising a taxi service including a telephone number; and processor 101 upon extracting the telephone number from the image may issue a dial command to a telephone 114 as illustrated in
For example, a user may aim a mobile device at a bus-stop or underground railway sign and obtain access to the time table of the public transportation system. This could include positioning in case the user cannot identify from the sign which bus stop he or she is pointing at. This could be useful in areas where the time tables are unreadable for some reason (e.g., dirty or damaged). This could also be useful in areas where the user is able to obtain information about bus or train delays from a website, allowing the user to choose between different alternatives.
The term “environmental cue” may include, among other things, a traffic sign or advertising sign contained in a received image; a radio signal comprising information relating to a location or other information such as telephone numbers; or current position information. Current position information may be obtained from a GPS receiver or other means.
Many variations of the inventive principles are possible. The method steps described herein can be implemented in computer software and encoded on computer-readable media for execution by a computer processor or specialized circuit, and the invention includes such computer-readable media.
One or more aspects of the invention may be embodied in computer-executable instructions, such as in one or more program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types when executed by a processor in a computer or other device. The computer executable instructions may be stored on a computer readable medium such as a hard disk, optical disk, removable storage media, solid state memory, RAM, etc. As will be appreciated by one of skill in the art, the functionality of the program modules may be combined or distributed as desired in various embodiments. In addition, the functionality may be embodied in whole or in part in firmware or hardware equivalents such as integrated circuits, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), and the like. The terms “processor” and “memory” comprising executable instructions should be interpreted individually and collectively to include the variations described in this paragraph and equivalents thereof.
Embodiments include any novel feature or combination of features disclosed herein either explicitly or any generalization thereof. While embodiments have been described with respect to specific examples including presently preferred modes of carrying out the invention, those skilled in the art will appreciate that there are numerous variations and permutations of the above described systems and techniques. Thus, the spirit and scope of the invention should be construed broadly as set forth in the appended claims.
Claims
1. A method on a mobile device, comprising:
- receiving a visual image of a sign captured by a camera module of the mobile device;
- obtaining position information of said mobile device;
- processing said visual image to identify an information element contained on said visual image; and
- accessing a timetable when said information element relates to a transportation service, wherein said timetable is relative to said position information.
2. The method of claim 1, further comprising:
- dialing a phone number when said information element relates to a phone number.
3. The method of claim 1, further comprising:
- accessing a web page when said information element relates to a website address.
4. The method of claim 1, wherein processing said visual image to identify said information element comprises:
- storing a cue database; and
- correlating said visual image to one or more cues of said activated cue database.
5. A mobile device, comprising:
- a display configured to display a graphical user interface;
- a camera module configured to capture visual images;
- a GPS receiver configured to determine a current geographic location of said mobile device; and
- a processor configured to: receive a visual image of a sign captured by said camera module of said mobile device; obtain said current geographic location of said mobile device from said GPS receiver; process said visual image to identify an information element contained on said visual image; and access a timetable when said information element relates to a transportation service, wherein said timetable is relative to said current geographic position.
6. The mobile device of claim 5, wherein said processor is further configured to dial a phone number when said information element relates to a phone number.
7. The mobile device of claim 5, wherein said processor is further configured to access a web page when said information element relates to a website address.
8. The mobile device of claim 5, further comprising a storage device, wherein said processor is further configured to, when processing said visual image to identify said information element:
- store a cue database in the storage device; and
- correlate the visual image to one or more cues of said cue database.
9. A non-transitory computer-readable medium storing executable instructions configured to, when executed, cause a processor to:
- receive a visual image of a sign captured by a camera module of a mobile device;
- obtain position information of said mobile device;
- process said visual image to identify an information element contained on said visual image; and
- access a timetable when said information element relates to a transportation service, wherein said timetable is relative to said position information.
10. The computer-readable medium of claim 9, wherein said executable instructions are further configured to cause said processor to dial a phone number when said information element relates to a phone number.
11. The computer-readable medium of claim 9, wherein the executable instructions are configured to cause said processor to access a web page when said information element relates to a website address.
12. The computer-readable medium of claim 9, wherein for processing said visual image to identify said information element, the executable instructions further cause said processor to:
- store a cue database; and
- correlate said visual image to one or more cues of said activated cue database.
Type: Application
Filed: Feb 21, 2020
Publication Date: Aug 20, 2020
Applicant: CONVERSANT WIRELESS LICENSING S.A.R.L. (Luxembourg)
Inventors: Martin BUECKER (The Hague), Maxim LOBKO (Cologne), Marc HOFFMANN (Dinslaken), Edmund COERSMEIER (Bochum)
Application Number: 16/797,662