Motion Assessment Using a Game Controller
A technique is disclosed for motion assessment of an individual using a game controller in connection with interactive media, e.g. a multiplayer game, such as Simon says, where game play is based, at least in part, upon coordinated or directed physical interaction of multiple players.
This application claims priority to U.S. Provisional Patent Application Ser. No. 61/043,696, filed 9 Apr. 2008, which application is incorporated herein in its entirety by this reference thereto.
FIELD OF THE INVENTIONEmbodiments of this invention relate to interactive media. More particularly, embodiments of this invention relate to motion assessment of an individual using a game controller in connection with interactive media.
BACKGROUND OF THE INVENTIONSimon says is a game for three or more players, most often children. One of the people is “it”—i.e. Simon. The others must do what Simon tells them to do when asked with a phrase beginning with “Simon says.” If Simon says “Simon says jump,” then the other players must jump. Those players who do not jump are out. However, if Simon says simply “jump,” without first saying “Simon says,” players do not jump. Those who do jump are out. In general, it is the spirit of the command, not the actions that matters. If Simon says “Simon says touch your toes,” players only have to show that they are trying to touch their toes. It is the ability to distinguish between valid and invalid commands, rather than physical ability, that matters.
It is Simon's task to try to get everyone out as quickly as possible, and it is every one else's job to stay in for as long as possible. The last of Simon's followers to stay in wins.
Simon says is an example of a game that is played by a group of people, where each player is present at the same place at the same time. Games of this type are therefore not amenable to distributed play, for example with players at various remote locations, because there is no known way of coordinating and assessing each player's movement, for example, in response to a command from Simon in the Simon says game.
Thus, there is a need in the art, for a technique that assesses user motion in multiplayer games, where such games require physical interaction of the users although the players are at different locations.
SUMMARY OF THE INVENTIONThe disadvantages associated with the prior art are overcome by embodiments of the present invention directed to a technique for motion assessment of an individual using a game controller in connection with interactive media. In an embodiment, the media may be a multiplayer game, such as Simon says, where game play is based, at least in part, upon coordinated or directed physical interaction of multiple players.
The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:
Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the invention. Accordingly, the exemplary embodiments of the invention described below are set forth without any loss of generality to, and without imposing limitations upon, the claimed invention.
An embodiment of the invention is directed to a technique for motion assessment of an individual using a game controller in connection with interactive media. In an embodiment, the media may be a multiplayer game, such as Simon says, where game play is based, at least in part, upon coordinated or directed physical interaction of multiple players.
A plurality of participant sites 12a-12n allow one or more participants 17a-17n to respond to such instruction pursuant to game play. Actions of the participants at these sites are assessed via a motion assessment device 15a-15n, as discussed above which, in this embodiment, includes a video camera 19a-19n. In this embodiment, participant site12n assesses motion with devices 34 that are worn on each participant's wrist.
In this embodiment, a game server 16 coordinates game play, scoring, and can also provide image and voice recognition functions and other capability. Other embodiments rely upon a game console for such functions, while still other embodiments do not rely upon a game console, but provide all image and display information directly to a game server. Further, it is not necessary for each participant to use the same motion assessment mechanism. Thus, some locations may employ hand-held devices, while others may employ video cameras, etc.
An embodiment of the invention uses a video camera, such as the Sony PlayStation Eye, which is a digital camera device that operates in conjunction with the PlayStation 3 video game console. Interaction of game players may also be captured with a game controller or other device. The video game console and/or a game server operates to record movements between players or any entity, as detected by the video camera, game controller, or other modality, and to compare those movements between people at remote locations, for example on other sides of the Internet, as well a detecting multiple individuals at a same location and tracking their motions separately pursuant to game play. Other embodiments of the invention contemplate any other video camera and game console or game console equivalent, such as the Sony EyeToy and PlayStation 2. Additionally, and as discussed above, other capture modalities may be used in conjunction with the video camera, such as a motion detector, which may be an ultrasonic motion detection device, a magnetic motion capture device, an RF or IR motion capture device, a pressure sensitive device, such as an active floor mat, a device that tracks gravitational forces and/or acceleration, or a device that is held in the hand or attached to the game player's body, such as the Sony SIXAXIS wireless controller, or the like. Further, a device specifically designed for capturing motion images in three dimensions may be used, such as an array multiple, spaced cameras, a laser guided device, or the like.
The voice information is routed to a voice processor 29 that identifies spoken commands, such as “Hands on head,” and looks up a pose for such command in a pose database 20. The voice information may also be routed to the various participant locations via a routing mechanism 31.
The video information is routed to a video processor 28, which performs the functions of accessing the pose database pursuant to operation of a pose capture module 21. The video processor also provides video information to the routing mechanism for viewing at each participant location. An image extraction module 27 operates under supervision of the video processor and in conjunction with a player database 22 to identify participants and, in conjunction with the pose capture module and pose database, to identify participant actions. Pose information from the pose database is provided to a compare module 26 under control of the video and/or voice processors, where a pose template is compared with an extracted image and a determination of accuracy, etc. is made with reference to standards stored in a metrics database 23. A result of the comparison is provided to a scoring module 25. Scoring results are both stored and retrieved from a score database 24 for display to the participants with regard to game play. The score results and image information are combined at a composite module 32 and thence routed to the game participants.
In an embodiment that implements the classic game of Simon says, kids (and/or others) play Simon says against one another. At least one game player, who may be a master, i.e. “it,” and who is chosen randomly or who is chosen by some other scheme, describes a motion to be undertaken by the other game players. The master's commands are routed to all participants, for example via a network connection. The master may then make some motion in front of the video camera, which is captured and analyzed by, for example the game console and/or game server. When other kids try to do that movement, those movements are also captured, for example, by a video camera, and a component of the game console and/or game server programming compares the captured movements against those of the master to determine whether or not the master's movement was mimicked correctly or close enough to qualify as a successful action. The action may also be judged on the timeliness of the participant response, e.g. who was first, how fast was the response, did a time-out occur before the action was completed? The system may also simply observe and determine whether the game participants executed the correct motion, based upon, for example, verbal commands from the master.
One method of tracking the participant comprises processing images captured by, for example, the video camera and identifying both the identity of the participant, if there are multiple participants at a location, by recognizing, for example, each participant's face; and by tracking movement each participant's head, arms, legs, and torso, to identify, for example, a pose of the participant's body for purposes of comparison with that of a master and/or an arc of movement of the participant's body, for example, to identify compliance of the participant with more complex movements.
In an embodiment, a microphone associated with the video camera or otherwise situated in connection with at least a master, may be provided to capture user commands and, for example, via voice recognition, translate such commands into signals that are used to look up a pre-recorded motion template. For example, if the master says “Simon says put your hands on your head,” then a voice recognition module translates this command into a signal that looks up a “hands on head” template in a pose database. The pose captured from each of the participants is compared to the template of the pose and compliance is then determined and recorded for each participant. In this embodiment, the master need not demonstrate the pose, but merely commands that a pose be assumed. While these commands are captured for purposes of template look-up, as described above, a mechanism, such as a network, is also provided for routing these commands to each participant to allow them to be heard at each location.
Further, while the video camera is used to capture images of the participant's for analysis by the game console and/or game server, the images may also be provided to the master, for example to provide comments upon observed poses and the like. Also, if a remote classroom setting is provided, the game console and/or game server may be programmed to determine if a participant is not present or has left the view of the camera. In this way, an embodiment of the invention may be used to take class attendance and determine if each participant is indeed participating. This aspect of the invention may be used, not only in a game setting, but also in an instructional or educational setting, as discussed below.
The game console and/or game server may include programming that performs such motion-capture video analysis. While face recognition or other schemes for identifying individuals within a group may be applied in those embodiments where multiple individuals participate at a single location, it will be appreciated by those skilled in the art that each individual in the group may wear an identification device, such as an article of clothing that is a particular color and/or that has a particular marking; or an electronic bracelet or other such signaling device may be worn by each individual. In those cases where a single individual is present at a location, it is only necessary to know the location, for example by a network address, to identify the individual for purposes of tracking the individual's movements and scoring the individual's execution of these movements. Other embodiments contemplate the use of voice recognition to identify participants.
Likewise, a group's reaction can be captured, analyzed and scored. Thus, if Simon says “Put your hands on your head,” the system could determine how many participants within a group reacted correctly. The group could be scored on this basis. In this way, groups of players could compete against other groups of players, e.g. the 4th grade class could compete against the 5th grade class, the boys could compete against the girls, the students could compete against the faculty, etc.
Another embodiment provides educational and/or tutorial activities, for example where a pre-recorded demonstration of a desired movement by a teacher or other instruction is presented to one or more session participants, rather than having a master participant in real-time, as in the Simon says example above. Such instruction may be, for example, how to execute a golf swing, a tennis stroke, or other complex motion. A participant in such session views a demonstration of the motion and/or other instructional information, for example, on a television or other display. A video camera or other device captures the participant's motions, analyzes and compares the captured motions with an ideal motion, and is then judged or graded on how well he mimicked the action that was demonstrated to him.
An embodiment of the invention provides a reference gesture that, for example, is generated by a live person, e.g. over the Internet, who performs an action. The action that the user is performing may be displayed to the other participants by a video clip of the person doing it, i.e. the system records the person demonstrating the gesture and then it shows it as a video clip. The camera can also capture such gestures from multiple angles, or multiple cameras can be used, such that a plurality of views from the plane of the camera can be combined to provide a 3D view that allows each participant to see all perspectives of what the action is, whether it is for a Simon says game, a golf swing or tennis serve analyzer, or for learning dance moves.
An embodiment of the invention provides each participant with the ability to speak and move an object. Thus, each participant can create a login based on, for example, a unique manipulation of a game controller.
An embodiment comprehends gesture recording, where the gesture occurs in free space. A scoring aspect of invention determines what tolerance a participant's gesture achieves relative to a reference gesture, e.g. a snapshot. The game console and/or game server include programming that is used to compare differences between a reference gesture which may be captured, e.g. from a 3D tracking device, from a camera, or a combination of both, or which may be pre-recorded, e.g. an expert gesture, and some change over time for some purpose. Depending on the change, results are provided, such as the person moved the wrong way; or in a golf swing, from the time they start the swing to the time they end the swing, there is some change over time. For example, with regard to a golf swing, the gesture, i.e. the swing would be decomposed into various critical elements, where certain parts of the motion are more important than other parts of the motion. Thus, an embodiment of the invention provides a weighting of the critical elements within a movement or gesture, such as a golf swing, and relative to a reference, a person's performance is scored based on the weighting of various elements within an actual captured image.
An embodiment provides a library of functions, such as various types of golf swing or tennis serves. Such motions are all pre-computed and the video camera is used to capture a person's execution relative to the reference in the library to identify how much the person's performance deviates from the idealized motion. Further, an expert could watch the person execute a movement in real time and provide instruction and performance evaluation based upon analysis of a captured movement relative to a reference movement. In this example, the expert may be an instructor at a remote location who is observing the person's performance with the video camera at the person's location via a network.
Likewise, several individuals can demonstrate a skill simultaneously and be judged and ranked accordingly. For example, there could be an on-line golf swing competition in which each individual is observed and graded by the system. Results and ranking would then be exhibited on each participant's display. A human expert may also participant in the judging activity.
An embodiment starts game play when a participant makes a particular gesture that is recognized by the system as a sign that game play is to commence. Such gesture can be specified in advance by the game developer, or it can be recorded by a participant, for example in the Simon says game, a participant, perhaps the master, would lift his hands above his head to start a game play session. This aspect of the invention thus concerns performing an action and determining whether or not the action was taken, e.g. did the participants commence game play?
One embodiment comprises a game in which an initial position is established for a player by another player, where the player who is being instructed must follow the instructions of the other player. The person following the instructions has a reference picture taken on the initial position, and then after a command is issued by the instructing player, a further image is taken or further information is gathered and compared against the reference picture in real time to determine if it was performed according to some critical performance characteristic, e.g. quickly enough, accurately enough, or should not have been performed at all. As with Simon says, one person may be the leader and other participants follow some form of instruction, imperative command, or illustration of an action, and the participants then respond accordingly.
An embodiment of the invention may also be used in games that employ objects, as well as individual participants. For example, Hullabaloo (Cranium, Inc, Seattle, Wash.) is a talking game that is played with different colored cards. The cards are placed on the floor and the game instructs the game players where to move, often entangling the players, one with the other. An embodiment of the invention, in this vein, uses physical objects at each participant's location that are represented icons on a display as well. The objects are placed on the floor and the participants physically move to the objects as instructed by the icons. The invention tracks the participants, as discussed above, over a network based system.
In addition to tracking the individual, game elements could be placed in a room and the system uses the video camera or other such expedient to know the position of the game elements as well. Thus, an embodiment provides for, and tracks, both the players who are in the room, where the system knows their position, and the game elements that are a part of the play. Another embodiment identifies objects that are in a room, such as a table or chair, and assigns them a role in a game. For example, a participant may be instructed to sit on the chair or to lie down on the rug.
While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications and equivalents. Therefore, the scope of the present invention should be determined not with reference to the above description but should, instead, be determined with reference to the appended claims, along with their full scope of equivalents. Any feature described herein, whether preferred or not, may be combined with any other feature described herein, whether preferred or not. In the claims that follow, the indefinite article “A”, or “An” refers to a quantity of one or more of the item following the article, except where expressly stated otherwise. The appended claims are not to be interpreted as including means-plus-function limitations, unless such a limitation is explicitly recited in a given claim using the phrase “means for.”
Claims
1. An apparatus for motion assessment of an individual in connection with interactive media, comprising:
- a plurality of participant locations;
- a video camera at each participant location for capturing movements of each participant at each said participant location; and
- a processor programmed to execute instructions for comparing said captured movements of each participant at each of said participant locations against any of a reference movement, an expected or elicited movement, or a movement of one or more other participants.
2. The apparatus of claim 1, said interactive media comprising:
- a multiplayer game in which game play is based, at least in part, upon coordinated or directed physical interaction of multiple players.
3. The apparatus of claim 1, said interactive media comprising:
- a game in which at least one said participant location comprises a master site from which one or more participants issue instructions and/or demonstrates movements, which are captured via a video camera at said master site and communicated to each other of said participant locations;
- said each other of said participant locations comprising game play sites at each of which one or more participants respond to said instructions pursuant to game play, and wherein actions of participants at said participant locations are captured via a video camera at a corresponding game play site and communicated to said processor and/or said master site.
4. The apparatus of claim 2, further comprising:
- a game server, in communication with said plurality of participant locations via a network, programmed to execute instructions for any of coordinating game play, scoring game play, and providing image and voice recognition functions.
5. The apparatus of claim 1, further comprising:
- at least one additional motion capture modality comprising any of an ultrasonic motion detection device, a magnetic motion capture device, an RF or IR motion capture device, a pressure sensitive device, a device that tracks gravitational forces and/or acceleration, a device that is held in the hand or attached to a participant's body, and a device for capturing motion images in three dimensions.
6. The apparatus of claim 1, said processor programmed to execute instructions for detecting multiple participants at a same participant location and tracking their motions separately pursuant to game play.
7. The apparatus of claim 1, said processor programmed to execute instructions for receiving and executing voice commands received from one or more participants via a microphone a said or more participant's locations.
8. The apparatus of claim 1, said processor programmed to execute instructions for any of evaluating, critiquing, ranking, and scoring said captured movements, for any of each individual participant, groups of participants, and pursuant to a competition between two or more participants.
9. The apparatus of claim 1, said processor programmed to execute instructions for identifying both the identity of a participant, if there are multiple participants at a location, by recognizing each participant's face; and by tracking movement each participant's head, arms, legs, and torso, to identify a pose of the participant's body for purposes of comparison with that of a master and/or an arc of movement of the participant's body to identify compliance of the participant with more complex movements.
10. The apparatus of claim 1, said processor programmed to execute instructions for determining if a participant is not present at a participant location or has left the view of a camera at said participant location.
11. The apparatus of claim 1, said processor programmed to execute instructions for performing any of motion-capture video analysis.
12. The apparatus of claim 1, said processor programmed to execute instructions for demonstrating a desired movement or other instruction to one or more participants, for analyzing and comparing captured motions of said one or more participants responsive to said demonstrating relative to an ideal motion, and for judging and/or grading how well said one or more participants mimicked the movement that was so demonstrated.
13. The apparatus of claim 12, wherein said movement is a complex movement; and wherein portions of said movement are weighted relative to their difficulty, and/or accuracy and/or precision required.
14. The apparatus of claim 1, said processor programmed to execute instructions for accessing a library of motion templates.
15. The apparatus of claim 1, said processor programmed to execute instructions for using physical objects at each participant location pursuant to interaction with said interactive media and/or for identifying and/or tracking physical objects are each participant location.
16. A computer implemented method for motion assessment of an individual in connection with interactive media, comprising the steps of:
- providing a plurality of participant locations;
- capturing movements of each participant at each said participant location with a video camera; and
- comparing said captured movements of each participant at each of said participant locations against any of a reference movement, an expected or elicited movement, or a movement of one or more other participants.
17. The method of claim 16, said interactive media comprising:
- a multiplayer game in which game play is based, at least in part, upon coordinated or directed physical interaction of multiple players.
18. The method of claim 16, said interactive media comprising:
- a game in which at least one said participant location comprises a master site from which one or more participants issue instructions and/or demonstrates movements, which are captured via a video camera at said master site and communicated to each other of said participant locations;
- said each other of said participant locations comprising game play sites at each of which one or more participants respond to said instructions pursuant to game play, and wherein actions of participants at said participant locations are captured via a video camera at a corresponding game play site and communicated to said processor and/or said master site.
19. An electronic storage medium containing computer instructions for executing a method for motion assessment of an individual in connection with interactive media, comprising the steps of:
- providing a plurality of participant locations;
- capturing movements of each participant at each said participant location with a video camera; and
- comparing said captured movements of each participant at each of said participant locations against any of a reference movement, an expected or elicited movement, or a movement of one or more other participants.
Type: Application
Filed: Apr 2, 2009
Publication Date: Oct 15, 2009
Inventor: Aaron Philip Brunstetter (Foster City, CA)
Application Number: 12/417,384
International Classification: A63F 9/24 (20060101);
