Object detection for an interactive human interface device
The presence and/or movement of objects about a central location is detected, e.g., by sensors that emit and detect radiation within zones about the device.
This application claims priority to and the benefits of U.S. provisional patent application Ser. No. 60/704,623, filed Aug. 2, 2005, the entire disclosure of which is incorporated herein by reference.
BACKGROUNDVideo games generally involve a user (or multiple users) interacting with a game console, whether the console be a personal computer, a stand-alone machine (such as in an arcade) or a dedicated hardware device. These interactions are typically captured and communicated through controllers, such as joysticks, knobs, buttons, a keyboard, or a mouse. Such controllers are particularly suited for games in which complex actions within the game (running, jumping, flying an airplane, etc.) are predicated on relatively simple user commands such as clicking a mouse, hitting a spacebar, moving a joystick or pulling a trigger.
However, many popular games (both arcade and home-based) operate on a different premise. Instead of receiving user instructions that are translated into a corresponding action in the game, the game itself requires the user to perform specific physical actions (often within certain spatial and/or temporal parameters) in response to instructions from the game. One such example is the “Dance Dance Revolution” (“DDR”) game by KONAMI Corporation of Tokyo, Japan. In the DDR game, the user positions herself on a “dance-pad” in front of a video screen. The game displays various “dance moves” on the screen, and the user imitates the moves on the dance-pad. The system physically senses each movement of the user and compares it to the move displayed on the screen. As the game progresses, the moves become more difficult and are presented to the user in more rapid succession.
Currently existing dance-pads consist of layers of MYLAR surrounding a conductive carbon material that acts as a physical button or switch. Due to the physical nature of their intended use, however, conventional dance-pads suffer from significant problems that include wear and eventual malfunction of the switch contacts. Use in certain environments (e.g., on carpeted surfaces) is also difficult, due to slippage and the build-up of static electricity. Furthermore, typical dance-pads are generally cumbersome to carry, and due to the social nature of the DDR game, users often wish to bring dance games (and therefore the controllers) to social events.
Similar problems exist with controllers for other games in which users physically mimic or act out instructions and the user's complex physical actions are sensed and communicated back to the game console. Games in which a user acts out musical performances, for example, often include mimicking a musician (e.g., air-guitar, air-drumming, etc.). Such games typically require the user to hold a controller (such as a mock guitar or drumsticks) that are wired to the console. The user's interaction with the controller is sensed and sent to the console. Like the conventional dance-pads, however, the controllers are prone to wear and malfunction due to the physical nature of their use and mechanical nature of the sensing technology used. What is needed, therefore, is a new video game technology that eliminates the mechanical sensor-based controller and provides a smaller, lighter controller that can accurately and rapidly sense the actions of the users and does not wear out with repeated use.
SUMMARY OF THE INVENTIONThe invention takes advantage of optical, capacitive, ultrasonic, audio, electric field or other sensing modalities to detect both temporal and spatial object placement in one or more segments surrounding a central point or area. This obviates the need for physical contact to be made with a switch in order to detect the presence and/or location of an object about the center area. By using non-physical detection modes (e.g., modes in which the object does not need to physically touch the device in order for detection to occur), the device suffers significantly less trauma during use, is not subject to movement based on contact with the user, and can be manufactured smaller and lighter than conventional controllers. This is especially the case for devices used to detect actions that exert considerable force on the device such as stomping, punching or banging—actions commonly found in interactive video games.
In a first aspect, the invention provides an electronic sensing device that includes sensors placed along an envelope or edge about a central location. Each sensor provides both a source of radiation and a detector for the radiation within a radial zone that originates at the sensor. In addition, the device includes circuitry that, based on the radiation detected by the sensors in each respective zone, generates a control signal. The control signal may, for example, indicate the location and/or presence of an object about the device, and in some embodiments in which the device also includes a signal emitter, may transmit the signal (using, for example, wireless protocols) to another device, such as a receiver.
The wireless protocols by which the emitter transmits the signals may use one of any frequency and modulation transmission techniques, including, for example, the IEEE 802 family of protocols, such as the 802.11 and/or 802.15 (Bluetooth) protocols. (As used herein, the term 802 protocol refers generally to any of these IEEE protocols, whether in use now or in the future. A current listing of 802 protocols is given at http://standards.ieee.org/wireless/.) The radiation may comprise, for example, infrared radiation, ultrasonic radiation, sound radiation, and/or radio-frequency radiation. In certain embodiments, the device also detects variations in an electric field about the device.
The sensors may be placed at any location about the device; however, in certain embodiments, the sensors are spaced about the perimeter of the device in such a manner that the zones are substantially triangular. Once such apparatus in which the device is generally octagonally shaped includes eight sensors placed about the device such that each sensor is associated with one edge of the device. The zones may be of any shape and size; however, in certain embodiments, each zone is configured to accommodate (and thus the sensors detect) the presence of a human body part such as a human foot.
The control signals may, for example, represent an interaction of a human with the device, such as the placement, removal, or movement of a body part within and/or among the zones. In some embodiments, the device also includes a substantially horizontal platform disposed above the central location.
In another aspect of the invention, a method for detecting the position of an object includes defining a plurality of radial zones about a central location, periodically introducing radiation into the zones, using radiation within a zone to detect the presence of an object in that zone, and upon detection of an object within an irradiated zone, generating a control signal indicative of the detection. The method may also include transmitting the control signal to a video game receiver at a console, thus allowing the console to display images and/or instructions indicative of or in response to the presence of the detected object. An indication (e.g., a visual and/or audio signal) may also be provided as confirmation of detection of the object.
In a third aspect, the invention provides a video game system that includes a console for providing user instructions (which may in some cases be based on a signal received from a video game controller) via a display device, sensors placed about a central location, where each sensor includes a source of radiation and a detector for detecting reflected radiation within a radial zone originating at the sensor, and circuitry responsive to the sensors for generating a control signal upon detection of the reflected radiation within the zones. The control signal is conformed to a video game controller signal, and the video game controller signal is transmitted to the video game console.
In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.
In some embodiments, the number of sensors 210 active at any time may be manipulated (either manually by the user or automatically by a game console) to alter the difficulty of the game. Initially, for example, the game may use only four sensors 210—front, back, left and right. As the user progresses through the game, however, additional sensors 210 may be activated such that six, and then possibly eight sensors 210 are active. In some cases, multiple sensors 210 may be assigned to one zone 240. For example, a simpler version of the game may only detect whether a user has placed her foot in front of the pad or in back of the pad. In such cases, even though the individual sensors 210 detect movement in their respective zones 240, any movement detected by the three sensors 210 along the back edge of the device 100 may provide a signal to the game controller that the user has properly placed her foot behind the device 100.
In one embodiment, the device 100 is octagonally shaped with sensing circuitry placed along each edge, which allows for the sensing of eight discrete zones 240 surrounding the device 100. However, it should be understood that any number of zones 240 and corresponding sensors 210 are possible, the number and configuration of which depend on the function and difficultly of the game and the required detection accuracy of a user's movements about the device 100.
Infrared transmitting and receiving elements may be incorporated within each of the eight sensor modules 210 placed about the device 100. In embodiments in which the sensor modules 210 include infrared light-emitting diodes (IR LEDs), each transmitter emits radiation in a pattern that defines its operational area 230. An object within the operational area 230 of any sensor 210 will reflect some of the optical transmission back to the sensor's receiving element, thus indicating the presence of an object within the zone 240 attributed to the area 230. Each of the zones 240 may, for example, represent a virtual spatial segment corresponding to a region in which an object can be detected by the nearest sensor 210. The object to be sensed (e.g., a foot) reflects the IR light transmitted by the sensor's IR LED and the reflection is received by the sensor's IR receiver module. These zones 240 may represent, in using the dance-pad game as an example, virtual counterparts to the physical dance-pad squares used in conventional systems. In such cases, the sensors 210 are used to determine if a user's actions are consistent with instructions from the game, and the signals generated by the sensors correspond to detection of objects in each zone 240—effectively emulating conventional dance-pad controllers used in dance-pad video games. For example, if the front sensor 210 detects an object, the invention will send an “up” signal to the connected video game console or computer. If the left sensor 210 detects an object, the invention will send a “left” signal to the connected video game console or computer, and so on.
The device may, in this and other embodiments, use paired IR transmitters and receivers to optically detect object placement within one of the surrounding segments. For example, the sensors may radiate pulsed IR light at 38 KHz from IR LEDs with a 20-degree beam width, and detect reflections from objects placed within the path of the IR light using corresponding (and conventional) IR receivers. In implementations using eight segments, the IR bursts commutate from position 1 to position 8, accounting for the eight sections surrounding the central unit. The bursts occur at approximately 10 millisecond intervals and each lasts for approximately one millisecond. Hence, approximately 38 cycles of a 38 kHz signal are radiated in each pulse burst, which is more than adequate to provide the three to ten 38 kHz cycles necessary to initialize the IR sensors. The length of the pulse burst may be altered as necessary to balance the power consumption of the device and the visibility of the LED indicators (described below).
The location and enclosure of the IR LEDs and the IR receiver circuitry may be altered according to various implementations; however, certain arrangements are more effective than others. To prevent reflection from the floor, for example, the IR LEDs and receivers may be tilted slightly upwards, i.e., away from the floor. In addition, the output of each of the eight IR detectors may be connected to a visible red LED by using, for example, the negative-going outputs of the IR receivers to cause corresponding red LEDs to blink (at a 10 Hz rate, for example) when a reflection is received. This visual indication may be used to confirm proper circuit functions, but is not necessary for the proper operation of the device.
The oscillator 505 also drives a one-shot timer 530 that produces a one-millisecond, positive-going output. When that output is HIGH, the diode DI is back-biased, allowing an R-C oscillator 535 to operate at approximately 38 kHz. Because the one-shot timer 530 pulses HIGH for one millisecond every 10 milliseconds, the R-C oscillator 535 is only activated for the same one millisecond, thus producing a one millisecond long signal of a 38 kHz square wave every 10 milliseconds. The 38 kHz signal serves as the carrier for the IR transmission later in the circuit. The output of oscillator 535 is isolated and inverted through a 74C14 hex inverter Schmitt trigger 545, which delivers a positive-going 38 kHz burst to the second of the two inputs of the eight dual AND gates 515, providing the second trigger signal. As a result, when an AND gate registers HIGH from the shift register 510 at the 100 Hz rate, it also registers HIGH from the R-C oscillator 535 at a rate of 38 kHz. Thus, each LED 525 is effectively turned on for a brief period, during which it strobes on and off at the faster 38 kHz rate.
The power supply may be regulated to +5V by a 78L05 linear regulator fed by a 9V battery 540. This regulator may be included because the output frequency of the 38 kHz R-C oscillator 535 is voltage-dependent and should operate on a clean, noise- and jitter-free 5V supply.
Still referring to
The output of each of the IR receiver modules may, in some embodiments, be connected to a red LED for testing purposes and to provide the user with visible feedback as to the functionality of one or more of the sensing modules. The IR receivers generally require 4.5 to 5 volts to operate properly. The supply line may be decoupled from the rest of the circuitry by a 20-ohm (minimum) resistor bypassed with a 100 μF electrolytic capacitor to account for the fact that the IR receivers are susceptible to small levels of 38 kHz signal on their supply terminals. The visible red LED anodes are tied together and connected to +5V via a common 300-ohm resistor.
In an embodiment using a microcontroller as shown in
Suitable source code in the C programming language to the firmware residing on the microcontroller for this example is as follows:
One alternative embodiment varies the carrier frequency from each of the eight IR LEDs and matched IR receiver. Thus, instead of pulsing the IR signal at 38 kHz from each of the eight IR LEDs to be detected by each of the 38 kHz IR receivers, the device may use multiple carrier frequencies (including, but not limited to, 30 kHz, 33 kHz, 36 kHz, 36.7 kHz, 38 kHz, 40 kHz, and/or 56 kHz frequencies) to provide an additional level of filtering and interference control between the various segments surrounding the central unit. By using a separate carrier frequency for each of the IR LEDs and matched IR receivers, the chance of a “false positive” detection from a sensor is reduced, as the receiver is tuned to only filter and receive data riding on its particular carrier frequency. As a result, a reflection from an object in an adjacent or different position will not be detected. One effective approach is to alternate the frequency of adjacent IR LEDs and receivers between widely separated frequencies, for example, 30 kHz and 56 kHz.
The invention may, in certain instances, use multiple sensors (in this example, IR LEDs and IR receivers) for each virtual zone in order to obtain a more accurate representation of a user's body part placement within a given zone around the center unit. As such, instead of using a single IR LED and IR receiver pair per zone, any number of IR LEDs and IR receivers may be used and will help to increase the accuracy of object placement within a particular zone.
Software-based techniques may also be employed on the microcontroller to aid in the accurate detection of an object within a particular zone surrounding the device. Such techniques may comprise, as examples, modifying the IR transmitting and IR receiving order of the sensors surrounding the sensor unit (for example, activating the sensors in an order different from a sequential clockwise or counter-clockwise motion around the central unit), or the use of signal processing to aid in filtering the signal received from any sensor to help ensure that the reflected signal received is the proper signal destined for that particular sensor (for example, to prevent the sensor from receiving a signal intended for another, possibly nearby, sensor).
Another embodiment of the present invention replaces the multiple IR LEDs and IR receivers with a single IR LED and receiver unit mounted onto a rotating arm or disc inside of the central unit. A motor drives the arm or disc in a clockwise or counterclockwise direction in an orientation parallel to the floor, commutating through the eight different positions. All of the necessary sensor electronics may be mounted onto the arm and the power input from the stationary base may be supplied via contacts to slip rings, also called rotary electrical joints.
In another embodiment, one or more patterns are projected onto the floor, each pattern (or segment thereof) corresponding to each of the segments surrounding the device. An optical detector co-located with the LED transmitting light sources normally receives light reflected from the floor when illuminated by these sequentially activated LEDs. In one form, the pattern comprises individual spots generated from a colored LED (for example, red to denote up/front, green to denote left, yellow to denote right, blue to denote down/back, and orange to denote the four corner locations). Adding such a visual indication of object detection can enhance the user experience; for example, the invention may blink or turn off the single spot generated from the LED corresponding to the segment that has detected an object, so the player knows the object was correctly sensed by the sensor. If the player puts a foot into the left position, for example, the green LED used to illuminate that particular area will be turned off to signify that the foot was correctly sensed.
In a similar embodiment in which LEDs corresponding to each zone are incorporated into the device, the LEDs may be used as status indicators which can be controlled by the video game console or other device or product in communication with the device. As such, the LEDs may be used as a training aid or as additional instructional elements. Using the dance-pad example, a DDR-type game may include a “beginner mode” in which flashing LEDs denote the zone into which the user should next step. The LEDs may also be used for numerous other interactive experiences, and the illumination of the LEDs may be controlled either by the device, a game console, or an external product interfacing thereto.
A delay may be added to the confirmed detection of an object by the sensing module. Since each of the sensing modules (comprising an IR LED and IR receiver module in this example) is, in certain embodiments, mounted slightly above the ground and angled upwards in order to prevent optical reflections from the floor, the object may be detected before it actually reaches the ground or floor. Thus, where the device is being used as a virtual dance pad or video game controller, a discrepancy can exist between the actual physical action and detection. Adding a small time delay between when the object is detected and when the object may be presumed to have reached the ground (e.g., approximately 10 milliseconds after detection) provides the user with feedback consistent with conventional controllers, in which the object must make physical contact with the controller's matrix switch connection located on the floor. This time delay can be added, for example, using a timer module internal to a general-purpose microcontroller, with a standard 555 timer module IC, or through software implemented on the microcontroller or other circuitry within the device.
In still another embodiment of the present invention, the IR receivers may be replaced by optical fibers, each of which guides one of the received reflected IR beams to a central location in the device. There, the IR light beam leaving each fiber is directed at a single, horizontally disposed IR receiver that is irradiated by reflected IR light picked up by the fiber and emitted at its opposite end. Additionally, each fiber's entry and exit geometry may be lensatic or supplemented by external lenses to increase the efficiency of light collection and delivery.
Another embodiment of the present invention uses optical lenses and/or light guides (fabricated, for example, of plastic) to control the optical emissions transmitted from the IR transmitter and/or to control the optical emissions being received by the IR receiver. Such optical lenses (or other type of lensatic or other optical control of a sensor's emission and reception characteristics) may be placed between the sensors and the exterior of the device to help guide the direction of the transmitting and receiving energy, to reduce interference with other nearby sensors and/or to provide a more accurate sensing of an object that reflects the transmitter's energy.
For example, controlling the displacement of IR light without using such a lens may, in certain circumstances, be difficult. Although one specification of an IR LED is its transmitting half-angle (for example, 20 degrees) and one specification of an IR receiver is its reception half-angle, further exterior lenses may be used to help guide the light even more accurately. Furthermore, by using lenses to guide the emitted and reflected light, the sensors may sense an object as low to the floor as possible, which is particularly important when the device replaces the conventional dance-mats that sense a foot-strike at floor level.
In another embodiment of the present invention, a capacitive or electric field sensor (for example, the Freescale MC34940 or MC33794 sensors) may be used to detect object positioning around the central unit. By using multiple electrodes located at various positions around the center unit (for example, eight), each electrode can detect a change in capacitance or electric field to determine if an object is in its detection range and, if so, the distance/proximity of the object and possibly the type of object (for example, distinguishing a human object from a non-living material such as a carpet or plastic toy).
It is to be understood that the references to eight sensor positions on the various embodiments of the device is for illustrative purposes only and is based on the use of an eight switch position controller in common, switch-matrix dance-pads. Any number of sensor positions, greater or fewer than eight, are also envisioned by the current invention. Further, any suitable type of sensing modality can be used with the present invention including, but not limited to, ultrasonic or audio sensing technologies. The description herein described infrared, capacitive, and electric field techniques as examples of the wide range of possible sensing modalities that may be implemented in the device.
Variations, modifications, and other implementations of what is described herein will occur to those of ordinary skill in the art without departing from the spirit and the scope of the invention as claimed. Accordingly, the invention is to be defined not by the preceding illustrative description but instead by the spirit and scope of the following claims.
Claims
1. An electronic sensing device comprising:
- a. a plurality of sensors disposed along an envelope about a central location, each sensor comprising a source of radiation and a detector therefor, the detector detecting reflected radiation within a radial zone originating at the sensor; and
- b. circuitry, responsive to the sensors, for generating a control signal upon detection of the reflected radiation within one or more of the zones.
2. The sensing device of claim 1 further comprising a platform disposed above the central location.
3. The sensing device of claim 2 further comprising a signal emitter for transmitting the control signal.
4. The sensing device of claim 3 wherein the signal emitter wirelessly transmits the control signal.
5. The sensing device of claim 4 wherein the wireless transmission utilizes one or more 802 wireless protocols.
6. The sensing device of claim 2 further comprising a receiver for receiving the transmitted control signals.
7. The sensing device of claim 1 wherein the radiation comprises infrared radiation.
8. The sensing device of claim 1 wherein the radiation comprises ultrasonic radiation.
9. The sensing device of claim 1 wherein the radiation comprises sound radiation.
10. The sensing device of claim 1 wherein the radiation comprises radio-frequency radiation.
11. The sensing device of claim 1 wherein the sensors further detect variations in an electric field about the device.
12. The sensing device of claim 1 wherein the sensors are spaced about the central location.
13. The sensing device of claim 12 wherein the sensors are spaced such that each radial zone is substantially triangular.
14. The sensing device of claim 12 wherein the device comprises eight sensors arranged octagonally.
15. The sensing device of claim 1 further comprising horizontal base, and wherein the sensors are angled upwardly relative to the base.
16. The sensing device of claim 1 wherein the zones are sized to accommodate a human foot.
17. The sensing device of claim 1 further comprising a sensitivity adjustment for altering the sensitivity of the sensors.
18. The sensing device of claim 1 wherein the control signal represents an interaction of a human with the device.
19. The sensing device of claim 18 wherein the interaction comprises the placement of a body part within one or more of the zones.
20. The sensing device of claim 19 wherein the body part comprises a foot.
21. A method of detecting position of an object, the method comprising:
- a. defining a plurality of radial zones about a central location;
- b. periodically introducing radiation into various ones of the zones;
- c. using radiation within a zone to detect the presence of an object in that zone; and
- d. upon detection of an object within an irradiated zone, generating a control signal indicative of the detection.
22. The method of claim 21 further comprising transmitting the control signal to a video receiver at a console, whereupon the console displays images indicative of the presence of the object in that zone.
23. The method of claim 21 wherein the detection is based at least in part on a sensitivity setting.
24. The method of claim 21 further comprising providing visual confirmation of a detection.
25. A video game system comprising:
- a. a video game console for providing user instructions via a display device;
- b. a plurality of sensors disposed along an envelope about a central location, each sensor comprising a source of radiation and a detector therefor, the detector detecting reflected radiation within a radial zone originating at the sensor;
- b. circuitry, responsive to the sensors, for generating a control signal upon detection of the reflected radiation within one or more of the zones, conforming the control signal to a video game controller signal, and transmitting the video game controller signal to the video game console.
26. The video game system of claim 25 wherein the instructions are based at least in part on the video game controller signal.
Type: Application
Filed: Aug 3, 2006
Publication Date: Feb 14, 2008
Inventors: Ralph H. Baer (Manchester, NH), Joseph B. Grand (San Diego, CA)
Application Number: 11/498,541
International Classification: A63F 9/24 (20060101);