Processing device with intuitive learning capability
A method and apparatus for providing learning capability to processing device, such as a computer game, educational toy, telephone, or television remote control, is provided to achieve one or more objective. For example, if the processing device is a computer game, the objective may be to match the skill level of the game with that of a player. If the processing device is an educational toy, the objective may be to increase the educational level of a user. If the processing device is a telephone, the objective may be to anticipate the phone numbers that a phone user will call. If the processing device is a television remote control, the objective may be to anticipate the television channels that will watched by the user. One of a plurality of actions (e.g., game actions, educational prompts, listed phone numbers, or listed television channels) to be performed on the processing device is selected. A user input indicative of a user action (e.g., a player action, educational input, called phone number, or watched television channel) is received. An outcome of the selected action and/or user action is determined. For example, in the case of a computer game, the outcome may indicate whether a computer-manipulated object has intersected a user-manipulated object. In the case of an educational toy, the outcome may indicate whether a user action matches a prompt generated by the educational toy. In the case of a telephone, the outcome may indicate whether a called phone number is on a list of phone numbers. In the case of a television remote control, the outcome may indicate whether a watched television channel is on a list of television channels. An action probability distribution that includes probability values corresponding to the plurality of actions is then updated based on the determined outcome. The next action will then be selected based on this updated action probability distribution. The foregoing steps can be modified based on a performance index to achieve the objective of the processing device so that it learns.
[0001] This application is a continuation-in-part of copending U.S. application Ser. No. 10/185,239, filed Jun. 26, 2002, which claims priority from U.S. Provisional Application Ser. No. 60/301,381, filed Jun. 26, 2001, U.S. Provisional Application Ser. No. 60/316,923, filed Aug. 31, 2001, and U.S. Provisional Application Ser. No. 60/378,255, filed May 6, 2002, all of which are hereby fully and expressly incorporated herein by reference.
COMPUTER PROGRAM LISTING APPENDIX[0002] A Computer Program Listing Appendix is filed herewith, which comprises an original compact disc (and duplicate compact disc containing the same) containing the MS Word files (1) “Intuition Intelligence-duckgame1.doc” of size 119 Kbytes, created on Aug. 23, 2001, and written in Visual Basic 6.0; (2) “Intuition Intelligence-duckgame2.doc” of size 119 Kbytes, created on Aug. 23, 2001, and written in Visual Basic 6.0; (3) “Intuition Intelligence-incomingphone.doc” of size 81 Kbytes, created on Apr. 28, 2002, and written in PHP; (4) “Intuition Intelligence-outgoing-phone.doc” of size 61 Kbytes, created on Apr. 28, 2002, and written in PHP; (5) “Intuition Intelligence-remote.doc” of size 129 Kbytes, created on Aug. 30, 2002, and written in Visual Basic 5.0; and (6) and “Intuition Intelligence-simonsays.doc” of size 60 Kbytes, created on Aug. 30, 2002, and written in Visual Basic 5.0. The Computer Program Listing Appendix is fully and expressly incorporated herein by reference.
TECHNICAL FIELD OF THE INVENTION[0003] The present inventions relate to methodologies for providing learning capability to processing devices, e.g., computers, microprocessors, microcontrollers, embedded systems, network processors, and data processing systems, and those products containing such devices.
BACKGROUND OF THE INVENTION[0004] The era of smart interactive computer-based devices has dawned. There is a demand to increasingly develop common household items, such as computerized games and toys, smart gadgets and home appliances, personal digital assistants (PDA's), and mobile telephones, with new features, improved functionality, and built-in intelligence and/or intuition, and simpler user interfaces. The development of such products, however, has been hindered for a variety of reasons, including high cost, increased processing requirements, speed of response, and difficulty of use.
[0005] For example, in order to attain a share in the computer market today, computer game manufacturers must produce games that are challenging and maintain the interest of players over a significant period of time. If not, the games will be considered too easy, and consumers as a whole will opt not to purchase such games. In order to maintain a player's interest in single-player games (i.e., the player plays against the game program), manufacturers design different levels of difficulty into the game program. As the player learns the game, thus improving his or her skill level, he or she moves onto the next level. In this respect, the player learns the moves and strategy of the game program, but the game program does not learn the moves and strategy of the player, but rather increases its skill level in discrete step. Thus, most of today's commercial computer games cannot learn or, at the most, have rudimentary learning capacity. As a result, player's interest in the computer game will not be sustained, since, once mastered, the player will no longer be interested in the game. Even if the computer games do learn, the learning process is generally slow, ineffective, and not instantaneous, and does not have the ability to apply what has been learned.
[0006] Even if the player never attains the highest skill level, the ability of the game program to change difficulty levels does not dynamically match the game program's level of play with the game player's level of play, and thus, at any given time, the difficulty level of the game program is either too low or too high for the game player. As a result, the game player is not provided with a smooth transition from novice to expert status. As for multi-player computer games (i.e., players that play against each other), today's learning technologies are not well understood and are still in the conceptual stage. Again, the level of play amongst the multiple players are not matched with other, thereby making it difficult to sustain the players' level of interest in the game.
[0007] As for PDA's and mobile phones, their user applications, which are increasing at an exponential rate, cannot be simultaneously implemented due to the limitation in memory, processing, and display capacity. As for smart gadgets and home appliances, the expectations of both the consumers and product manufacturers that these new advanced products will be easier to use have not been met. In fact, the addition of more features in these devices has forced the consumer to read and understand an often-voluminous user manual to program the product. Most consumers find it is extremely hard to understand the product and its features, and instead use a minimal set of features, so that they do not have to endure the problem of programming the advanced features. Thus, instead of manufacturing a product that adapts to the consumers' needs, the consumers have adapted to a minimum set of features that they can understand.
[0008] Audio/video devices, such as home entertainment systems, provide an added dimension of problems. A home entertainment system, which typically comprises a television, stereo, audio and video recorders, digital videodisc player, cable or satellite box, and game console is commonly controlled by a single remote control or other similar device. Because individuals in a family typically have differing preferences, however, the settings of the home entertainment system must be continuously reset through the remote control or similar device to satisfy the preferences of the particular individual that is using the system at the time. Such preferences may include, e.g., sound level, color, choice of programs and content, etc. Even if only a single individual is using the system, the hundreds of television channels provided by satellite and cable television providers makes it difficult for such individual to recall and store all of his or her favorite channels in the remote control. Even if stored, the remote control cannot dynamically update the channels to fit the individual's ever changing preferences.
[0009] To a varying extent, current learning technologies, such as artificial intelligence, neural networks, and fuzzy logic, have attempted to solve the afore-described problems, but have been generally unsuccessful because they are either too costly, not adaptable to multiple users (e.g., in a family), not versatile enough, unreliable, exhibit a slow learning capability, require too much time and effort to design into a particular product, require increased memory, or cost too much to implement. In addition, learning automata theory, whereby a single unique optimum action is to be determined over time, has been applied to solve certain problems, e.g., economic problems, but have not been applied to improve the functionality of the afore-mentioned electronic devices. Rather, the sole function of the processing devices incorporating this learning automata theory is the determination of the optimum action.
[0010] There, thus, remains a need to develop an improved learning technology for processors.
SUMMARY OF THE INVENTION[0011] The present inventions are directed to an enabling technology that utilizes sophisticated learning methodologies that can be applied intuitively to improve the performance of most computer applications. This enabling technology can either operate on a stand-alone platform or co-exist with other technologies. For example, the present inventions can enable any dumb gadget/device (i.e., a basic device without any intelligence or learning capacity) to learn in a manner similar to human learning without the use of other technologies, such as artificial intelligence, neural networks, and fuzzy logic based applications. As another example, the present inventions can also be implemented as the top layer of intelligence to enhance the performance of these other technologies.
[0012] The present inventions can give or enhance the intelligence of almost any product. For example, it may allow a product to dynamically adapt to a changing environment (e.g., a consumer changing style, taste, preferences, and usage) and learn on-the-fly by applying efficiently what it has previously learned, thereby enabling the product to become smarter, more personalized, and easier to use as its usage continues. Thus, a product enabled with the present inventions can self-customize itself to its current user or each of a group of users (in the case of multiple-users), or can program itself in accordance with a consumer's needs, thereby eliminating the need for the consumer to continually program the product. As further examples, the present inventions can allow a product to train a consumer to learn more complex and advanced features or levels quickly, can allow a product to replicate or mimic the consumer's actions, or can assist or advise the consumer as to which actions to take.
[0013] The present inventions can be applied to virtually any computer-based device, and although the mathematical theory used is complex, the present inventions provide an elegant solution to the foregoing problems. In general, the hardware and software overhead requirements for the present inventions are minimal compared to the current technologies, and although the implementation of the present inventions within most every products takes very little time, the value that they add to a product increases exponentially.
[0014] In accordance with a first aspect of the present inventions, a method of providing learning capability to a processing device comprises receiving an action performed by a user, and selecting one of a plurality of processor actions. By way of non-limiting example, the processing device can be a computer game, in which case, the user action can be a player move, and the processor actions can be game moves. Or the processing device can be an educational toy, in which case, the user action can be a child action, and the processor actions and be toy actions. Or the processing device can be a telephone system, in which case, the user action can be a called phone number, and the processor actions can be listed phone numbers. Or the processing device can be a television channel control system, in which case, the user action can be a watched television channel, and the processor actions can be listed television channels. The processing device can be operated in a single user environment, multiple user environment, or both. The processor action can be selected in response to the received user action or in response to some other information or event.
[0015] In any event, the processor action selection is based on an action probability distribution that contains a plurality of probability values corresponding to the plurality of processor actions. For example, the selected processor action can correspond to the highest probability value within the action probability distribution, or can correspond to a pseudo-random selection of a value within the action probability distribution. The action probability distribution may be initially generated with equal probability values (e.g., if it is not desired that the processing device learn more quickly of if no assumptions are made as to which processor actions are more likely to be selected in the near future) or unequal probability values (if it is desired that the processing device learn more quickly, and if it is assumed that there are certain processor actions that are more likely to be selected in the near future). Preferably, the action probability distribution is normalized.
[0016] The method further comprises determining an outcome of one or both of the received user action and selected processor action. By way of non-limiting example, the outcome can be represented by one of two values (e.g., zero if outcome is not successful, and one if outcome is successful), one of a finite range of real numbers (e.g., higher numbers may mean outcome is more successful), or one of a range of continuous values (e.g., the higher the number, the more successful the outcome may be). It should be noted that the outcome can provide an indication of events other than successful and unsuccessful events. If the outcome is based thereon, the selected processor action can be a currently selected processor action, previously selected processor action (lag learning), or subsequently selected processor action (lead learning).
[0017] The method further comprises updating the action probability distribution based on the outcome. A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device. The action probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0018] Lastly, the method comprises modifying one or more of the processor action selection, outcome determination, and action probability distribution update steps based on the objective(s). This modification can be performed, e.g., deterministically, quasi-deterministically, or probabilistically. It can be performed using, e.g., artificial intelligence, expert systems, neural networks, fuzzy logic, or any combination thereof. These steps can be modified in any combination of a variety of ways. For example, one of a predetermined plurality of algorithms used when updating the action probability distribution can be selected. One or more parameters within an algorithm used when updating the action probability distribution can be selected. The action probability distribution, itself, can be modified or transformed. Selection of an action can be limited to or expanded to a subset of probability values contained within the action probability distribution. The nature of the outcome or otherwise the algorithms used to determine the outcome can be modified.
[0019] Optionally, the method may further comprise determining a performance index indicative of a performance of the processing device relative to one or more objectives of the processing device, wherein the modification is based on the performance index. The performance index may be updated when the outcome is determined, and may be derived either directly or indirectly from the outcome. The performance index can even be derived from the action probability distribution. The performance index may be an instantaneous value or a cumulative value.
[0020] In accordance with a second aspect of the present inventions, a processing device comprises a probabilistic learning module having a learning automaton configured for learning a plurality of processor actions in response to a plurality of actions performed by a user, and an intuition module configured for modifying a functionality of the probabilistic learning module based on one or more objectives of the processing device, e.g., by selecting one of a plurality of algorithms used by the learning module, or modifying a parameter of an algorithm employed by the learning module. The processing device can be operated in a single user, multiple user environment, or both. Optionally, the intuition module can be further configured for determining a performance index indicative of a performance of the probabilistic learning module relative to the objective(s), and for modifying the probabilistic learning module functionality based on the performance index. The intuition module can be, e.g., deterministic, quasi-deterministic, or probabilistic. It can use, e.g., artificial intelligence, expert systems, neural networks, or fuzzy logic.
[0021] In the preferred embodiment, the probabilistic learning module may include an action selection module configured for selecting one of a plurality of processor actions. The action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The probabilistic learning module may further comprise an outcome evaluation module configured for determining an outcome of either or both of the received user action and selected processor action. The probabilistic learning module may further comprise a probability update module configured for updating the action probability distribution based on the outcome. When modifying the functionality of the learning module, the intuition module may modify a functionality of any combination of the action selection module, outcome evaluation module, and probability update module.
[0022] In accordance with a third aspect of the present inventions, a method of providing learning capability to a computer game is provided. One of the objectives of the computer game is to match the skill level of the computer game with the skill level of the game player. The method comprises receiving a move performed by the game player, and selecting one of a plurality of game moves. The computer game can be operated in either a single player environment, multiple player environment, or both. The game move can be selected in response to the received player move or in response to some other information or event.
[0023] In one preferred method, the plurality of game moves is performed by a game-manipulated object that can be visual to the game player, such as a duck, and the player move is performed by a user-manipulated object that is visual to the game player, such as a gun. In this case, the plurality of game moves can be discrete movements of the game-manipulated object. Alternatively, the plurality of game moves can be delays related to a movement of the game-manipulated object. The player move can be a simulated shot taken by the user-manipulated object.
[0024] In any event, the game move selection is based on a game move probability distribution that contains a plurality of probability values corresponding to the plurality of game moves. For example, the selected game move can correspond to the highest probability value within the game move probability distribution, or can correspond to a pseudo-random selection of a value within the game move probability distribution. The game move probability distribution may be initially generated with equal probability values (e.g., if it is not desired that the computer game learn more quickly of if no assumptions are made as to which game moves are more likely to be selected in the near future) or unequal probability values (if it is desired that the computer game learn more quickly, and if it is assumed that there are certain game moves that are more likely to be selected in the near future).
[0025] The method further comprises determining an outcome of the received player move and selected game move. By way of non-limiting example, the outcome can be determined by performing a collision technique on the player move and selected game move. For example, the outcome can be represented by one of only two values, e.g., zero (occurrence of collision) and one (non-occurrence of collision), one of a finite range of real numbers (higher numbers mean lesser extent of collision), or one of a range of continuous values (the higher the number, the less the extent of the collision). The outcome is determined by performing a collision technique on the player move and the selected game move. If the outcome is based thereon, the selected game move can be a currently selected game move, previously selected game move (lag learning), or subsequently selected game move (lead learning).
[0026] The method further comprises updating the game move probability distribution based on the outcome. A learning automaton can optionally be utilized to update the game move probability distribution. A learning automaton can be characterized in that any given state of the game move probability distribution determines the state of the next game move probability distribution. That is, the next game move probability distribution is a function of the current game move probability distribution. Advantageously, updating of the game move probability distribution using a learning automaton is based on a frequency of the game moves and/or player moves, as well as the time ordering of these game moves. This can be contrasted with purely operating on a frequency of game moves or player moves, and updating the game move probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the computer game. The game move probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0027] Lastly, the method comprises modifying one or more of the game move selection, outcome determination, and game move probability distribution update steps based on the objective of matching the skill levels of the game player and computer game. These steps can be modified in any combination of a variety of ways. For example, one of a predetermined plurality of algorithms used when updating the game move probability distribution can be selected. One or more parameters within an algorithm used when updating the game move probability distribution can be selected. The game move probability distribution, itself, can be modified or transformed. Selection of a game move can be limited to or expanded to a subset of probability values contained within the game move probability distribution. The nature of the outcome or otherwise the algorithms used to determine the outcome can be modified.
[0028] In the preferred embodiment, if the game move selection is modified, the plurality of game moves can be organized into a plurality of game move subsets, and the game move can be selected from one of the plurality of game move subsets. A subsequent game move selection will then comprise selecting another game move subset from which a game move can be selected.
[0029] The method may optionally comprise determining a performance index indicative of a performance of the computer game relative to the objective of matching the skill levels of the computer game and game player (e.g., a relative score value between the computer game and the game player), wherein the modification is based on the performance index. The performance index may be updated when the outcome is determined, and may be derived either directly or indirectly from the outcome. The performance index can even be derived from the game move probability distribution. The performance index may be an instantaneous value or a cumulative value.
[0030] In accordance with a fourth aspect of the present inventions, a computer game comprises a probabilistic learning module having a learning automaton configured for learning a plurality of game moves in response to a plurality of moves performed by a player. The game moves and player moves can be represented by game-manipulated objects and user-manipulated objects, as previously discussed. The computer game can be operated in either a single player environment, multiple player environment, or both. The computer game further comprises an intuition module configured for modifying a functionality of the probabilistic learning module based on an objective of matching the skill level of the computer game with the skill level of the game player, e.g., by selecting one of a plurality of algorithms used by the learning module, or modifying a parameter of an algorithm employed by the learning module. Optionally, the intuition module can be further configured for determining a performance index indicative of a performance of the probabilistic learning module relative to the objective of matching the skill level of the computer game with the skill level of the game player (e.g., a relative score value between the computer game and the game player), and for modifying the probabilistic learning module functionality based on the performance index.
[0031] In the preferred embodiment, the probabilistic learning module may include a game move selection module configured for selecting one of a plurality of game moves. The game move selection can be based on a game move probability distribution comprising a plurality of probability values corresponding to the plurality of game moves. The probabilistic learning module may further comprise an outcome evaluation module configured for determining an outcome of the received player move and selected game move. The probabilistic learning module may further comprise a probability update module configured for updating the game move probability distribution based on the outcome. When modifying the functionality of the learning module, the intuition module may modify a functionality of any combination of the game move selection module, outcome evaluation module, and probability update module.
[0032] In accordance with a fifth aspect of the present inventions, a method of providing learning capability to a processing device comprises generating an action probability distribution comprising a plurality of probability values organized among a plurality of action subsets, wherein the plurality of probability values correspond to a plurality of processor actions. The action subset may be, e.g., selected deterministically, quasi-deterministically, or probabilistically.
[0033] The method further comprises selecting one of the plurality of action subsets, and selecting (e.g., pseudo-randomly) one of a plurality of processor actions from the selected action subset. By way of non-limiting example, the selected action subset can correspond to a series of probability values within the action probability distribution. For example, the selected action subset can correspond to the highest probability values, lowest probability values, or middlemost probability values. The selected action subset can correspond to probability values, the average of which is relative (greater, less than or equal) to a threshold value (e.g., a median probability value) that can be fixed or dynamically adjusted. Optionally, the method comprises determining a performance index indicative of a performance of the processing device relative to one or more objectives, wherein the action subset selection is based on the performance index.
[0034] Depending upon the application of the processing device, the method may further comprise receiving an action performed by a user, determining an outcome of either or both of the received user action and selected processor action, and updating the action probability distribution based on the outcome. The processing device can be operated in a single user environment, multiple user environment, or both. The processor action may be selected in response to the received user action or in response to some other information or event.
[0035] The action probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device.
[0036] In accordance with a sixth aspect of the present inventions, a method of providing learning capability to a computer game comprises generating a game move probability distribution comprising a plurality of probability values organized among a plurality of game move subsets, wherein the plurality of probability values correspond to a plurality of game moves. The game move subset may be, e.g., selected deterministically, quasi-deterministically, or probabilistically.
[0037] The method further comprises selecting one of the plurality of game move subsets, and selecting (e.g., pseudo-randomly) one of a plurality of game moves from the selected game move subset. The game move subset can be selected in a variety of manners, as previously discussed above. In one preferred method, the game move subset can be selected based on a skill level of a game player relative to a skill level of the computer game obtained from, e.g., the difference between a game player score and a computer game score. In this case, the game move subset can be selected to correspond to the highest probability values within the game move probability distribution if the relative skill level is greater than a threshold level, to the lowest probability values within the game move probability distribution if the relative skill level is less than a threshold level, and to the middlemost probability values within the game move probability distribution if the relative skill level is within a threshold range. Alternatively, the game move subset can be selected to correspond to probability values having an average greater than a threshold level if the relative skill level value is greater than a relative skill threshold level, less than a threshold level if the relative skill level value is less than a relative skill threshold level, or substantially equal to a threshold level if the relative skill level value is within a relative skill threshold range.
[0038] The method may further comprise receiving a player move, determining an outcome of the received player move and selected game move, and updating the game move probability distribution based on the outcome. The computer game can be operated in either a single player environment, multiple player environment, or both. The game move may be selected in response to the received player move or in response to some other information or event. In one preferred method, the plurality of game moves is performed by a game-manipulated object that can be visual to the game player, such as a duck, and the player move is performed by a user-manipulated object that is visual to the game player, such as a gun. In this case, the plurality of game moves can be discrete movements of the game-manipulated object. Alternatively, the plurality of game moves can be delays related to a movement of the game-manipulated object. The player move can be a simulated shot taken by the user-manipulated object.
[0039] In accordance with a seventh aspect of the present inventions, a method of providing learning capability to a processing device generating a game move probability distribution using one or more learning algorithms, modifying the learning algorithm(s), and updating the game move probability distribution using the modified learning algorithm(s). The game move probability distribution comprises a plurality of probability values corresponding to a plurality of game moves. The learning algorithm(s) may be, e.g., modified deterministically, quasi-deterministically, or probabilistically. The learning methodologies can be any combination of a variety of types, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0040] In one preferred embodiment, one or more parameters of the learning algorithm(s) (e.g., a reward parameter, penalty parameter, or both) are modified. For example, one or both of the reward and penalty parameters can be increased, decreased, negated, etc. based on a function. Optionally, the method comprises determining a performance index indicative of a performance of the processing device relative to one or more objectives, wherein the learning algorithm modification is based on the performance index.
[0041] Depending upon the application of the processing device, the method may further comprise receiving an action performed by a user, determining an outcome of either or both of the received user action and selected processor action, and updating the action probability distribution based on the outcome. The processing device can be operated in a single user environment, multiple user environment, or both. The processor action may be selected in response to the received user action or in response to some other information or event.
[0042] The action probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device.
[0043] In accordance with an eighth aspect of the present inventions, a method of providing learning capability to a computer game comprises generating a game move probability distribution using one or more learning algorithms, modifying the learning algorithm(s), and updating the game move probability distribution using the modified learning algorithm(s). The learning algorithm(s) can be similar to those previously discussed above, and can be modified in a manner similar to that described above.
[0044] In one preferred method, the learning algorithm modification can be based on a skill level of a game player relative to a skill level of the computer game obtained from, e.g., the difference between a game player score and a computer game score. In this case, the learning algorithm(s) can be modified by increasing a reward and/or penalty parameter if the relative skill level is greater than a threshold level, or decreasing or negating the reward and/or penalty parameter if the relative skill level is less than a threshold level.
[0045] The method may further comprise receiving a player move, determining an outcome of the received player move and selected game move, and updating the game move probability distribution based on the outcome. The computer game can be operated in either a single player environment, multiple player environment, or both. The game move may be selected in response to the received player move or in response to some other information or event. In one preferred method, the plurality of game moves is performed by a game-manipulated object that can be visual to the game player, such as a duck, and the player move is performed by a user-manipulated object that is visual to the game player, such as a gun. In this case, the plurality of game moves can be discrete movements of the game-manipulated object. Alternatively, the plurality of game moves can be delays related to a movement of the game-manipulated object. The player move can be a simulated shot taken by the user-manipulated object.
[0046] In accordance with a ninth aspect of the present inventions, a method of providing learning capability to a computer game is provided. One of the objectives of the computer game is to match the skill level of the computer game with the skill level of the game player. The method comprises receiving a move performed by the game player, and selecting one of a plurality of game moves. The computer game can be operated in either a single player environment, multiple player environment, or both. The game move can be selected in response to the received player move or in response to some other information or event. In any event, the game move selection is based on a game move probability distribution that contains a plurality of probability values corresponding to the plurality of game moves.
[0047] In one preferred method, the plurality of game moves is performed by a game-manipulated object that can be visual to the game player, such as a duck, and the player move is performed by a user-manipulated object that is visual to the game player, such as a gun. In this case, the plurality of game moves can be discrete movements of the game-manipulated object. Alternatively, the plurality of game moves can be delays related to a movement of the game-manipulated object. The player move can be a simulated shot taken by the user-manipulated object.
[0048] The method further comprises determining if the selected game move is successful, and determining a current skill level of the game player relative to a current skill level of the computer game obtained from, e.g., the difference between a game player score and a computer game score. By way of non-limiting example, the relative skill level can be determined to be relatively high if greater than a first threshold value, and relatively low if lower than a second threshold value.
[0049] Lastly, the method comprises updating the game move probability distribution using a reward and/or penalty algorithm based on the success of the selected game move and relative skill level. For example, the game move probability distribution can be updated using a reward algorithm if the selected game move is successful and the relative skill level is relatively high, or if the selected game move is unsuccessful and the relative skill level is relatively low; and/or the game move probability distribution can be updated using a penalty algorithm if the selected game move is unsuccessful and the relative skill level is relatively high, or if the selected game move is successful and the relative skill level is relatively low. Optionally, the reward algorithm and/or penalty algorithm can be modified based on the successful game move determination.
[0050] The game move probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the game move probability distribution determines the state of the next game move probability distribution. That is, the next game move probability distribution is a function of the current game move probability distribution. Advantageously, updating of the game move probability distribution using a learning automaton is based on a frequency of the game moves and/or player moves, as well as the time ordering of these game moves. This can be contrasted with purely operating on a frequency of game moves or player moves, and updating the game move probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the computer game.
[0051] In accordance with a tenth aspect of the present inventions, a method of providing learning capability to a computer game is provided. One of the objectives of the computer game is to match the skill level of the computer game with the skill level of the game player. The method comprises receiving a move performed by the game player, and selecting one of a plurality of game moves. The computer game can be operated in either a single player environment, multiple player environment, or both. The game move can be selected in response to the received player move or in response to some other information or event. In any event, the game move selection is based on a game move probability distribution that contains a plurality of probability values corresponding to the plurality of game moves.
[0052] In one preferred method, the plurality of game moves is performed by a game-manipulated object that can be visual to the game player, such as a duck, and the player move is performed by a user-manipulated object that is visual to the game player, such as a gun. In this case, the plurality of game moves can be discrete movements of the game-manipulated object. Alternatively, the plurality of game moves can be delays related to a movement of the game-manipulated object. The player move can be a simulated shot taken by the user-manipulated object.
[0053] The method further comprises determining if the selected game move is successful, and determining a current skill level of the game player relative to a current skill level of the computer game obtained from, e.g., the difference between a game player score and a computer game score. By way of non-limiting example, the relative skill level can be determined to be relatively high if greater than a first threshold value, and relatively low if lower than a second threshold value.
[0054] Lastly, the method comprises generating a successful outcome (e.g., “1” or “0”) or an unsuccessful outcome (e.g., “0” or “1”) based on the success of the selected game move and the relative skill level, and updating the game move probability distribution based on the generated successful outcome or unsuccessful outcome. For example, a successful outcome can be generated if the selected game move is successful and the relative skill level is relatively high, or if the selected game move is unsuccessful and the relative skill level is relatively low; and/or an unsuccessful outcome can be generated if the selected game move is unsuccessful and the relative skill level is relatively high, or if the selected game move is successful and the relative skill level is relatively low. Optionally, the reward algorithm and/or penalty algorithm can be modified based on the successful game move determination.
[0055] The game move probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the game move probability distribution determines the state of the next game move probability distribution. That is, the next game move probability distribution is a function of the current game move probability distribution. Advantageously, updating of the game move probability distribution using a learning automaton is based on a frequency of the game moves and/or player moves, as well as the time ordering of these game moves. This can be contrasted with purely operating on a frequency of game moves or player moves, and updating the game move probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the computer game.
[0056] In accordance with an eleventh aspect of the present inventions, a method of providing learning capability to a processing device comprises generating an action probability distribution comprising a plurality of probability values corresponding to a plurality of processor actions, and transforming the action probability distribution. The action probability distribution transformation may, e.g., be performed deterministically, quasi-deterministically, or probabilistically. By way of non-limiting example, the action probability distribution transformation may comprise assigning a value to one or more of the plurality of probability values, switching a higher probability value and a lower probability value, or switching a set of highest probability values and a set lowest probability values.
[0057] Optionally, the method comprises determining a performance index indicative of a performance of the processing device relative to one or more objectives, wherein the action probability distribution transformation is based on the performance index. Depending upon the application of the processing device, the method may further comprise receiving an action performed by a user, determining an outcome of either or both of the received user action and selected processor action, and updating the action probability distribution based on the outcome. The action probability distribution is updated prior to transforming it. The processing device can be operated in a single user environment, multiple user environment, or both. The processor action may be selected in response to the received user action or in response to some other information or event.
[0058] The action probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device.
[0059] In accordance with a twelfth aspect of the present inventions, a method of providing learning capability to a computer game comprises generating a game move probability distribution comprising a plurality of probability values corresponding to a plurality of game moves, and transforming the game move probability distribution. The game move probability distribution transformation may performed in a manner similar to that described above.
[0060] In one preferred method, the game move probability distribution transformation may be performed based on a skill level of a game player relative to a skill level of the computer game obtained from, e.g., the difference between a game player score and a computer game score. In this case, the game move probability distribution transformation can comprise switching a higher probability value and a lower probability value, or switching a set of highest probability values and a set of lowest probability values if the relative skill level, if the relative skill level is greater than or less than a threshold level.
[0061] The method may further comprise receiving a player move, determining an outcome of the received player move and selected game move, and updating the game move probability distribution based on the outcome. The game move probability distribution is updated prior to transforming it. The computer game can be operated in either a single player environment, multiple player environment, or both. The game move may be selected in response to the received player move or in response to some other information or event. In one preferred method, the plurality of game moves is performed by a game-manipulated object that can be visual to the game player, such as a duck, and the player move is performed by a user-manipulated object that is visual to the game player, such as a gun. In this case, the plurality of game moves can be discrete movements of the game-manipulated object. Alternatively, the plurality of game moves can be delays related to a movement of the game-manipulated object. The player move can be a simulated shot taken by the user-manipulated object.
[0062] In accordance with a thirteenth aspect of the present inventions, a method of providing learning capability to a processing device comprises generating an action probability distribution comprising a plurality of probability values corresponding to a plurality of processor actions, and limiting one or more of the probability values. The action probability limitation may, e.g., be performed deterministically, quasi-deterministically, or probabilistically. By way of non-limiting example, the action probability limitation may comprise limiting the probability value(s) to a high value and/or a low value.
[0063] Optionally, the method comprises determining a performance index indicative of a performance of the processing device relative to one or more objectives, wherein the action probability limitation is based on the performance index. Depending upon the application of the processing device, the method may further comprise receiving an action performed by a user, determining an outcome of either or both of the received user action and selected processor action, and updating the action probability distribution based on the outcome. The processing device can be operated in a single user environment, multiple user environment, or both. The processor action may be selected in response to the received user action or in response to some other information or event.
[0064] The action probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device.
[0065] In accordance with a fourteenth aspect of the present inventions, a method of providing learning capability to a computer game comprises generating a game move probability distribution comprising a plurality of probability values corresponding to a plurality of game moves, and limiting one or more of the probability values. The game move probability limitation may performed in a manner similar to that described above. In one preferred method, the game move probability distribution limitation may be performed based on a skill level of a game player relative to a skill level of the computer game obtained from, e.g., the difference between a game player score and a computer game score.
[0066] The method may further comprise receiving a player move, determining an outcome of the received player move and selected game move, and updating the game move probability distribution based on the outcome. The computer game can be operated in either a single player environment, multiple player environment, or both. The game move may be selected in response to the received player move or in response to some other information or event. In one preferred method, the plurality of game moves is performed by a game-manipulated object that can be visual to the game player, such as a duck, and the player move is performed by a user-manipulated object that is visual to the game player, such as a gun. In this case, the plurality of game moves can be discrete movements of the game-manipulated object. Alternatively, the plurality of game moves can be delays related to a movement of the game-manipulated object. The player move can be a simulated shot taken by the user-manipulated object.
[0067] In accordance with a fifteenth aspect of the present inventions, a method of providing learning capability to a processing device comprises receiving an action performed by a user, and selecting one of a plurality of processor actions. The processing device can be, e.g., a computer game, in which case, the user action can be a player move, and the processor actions can be game moves. Or the processing device can be a telephone system, in which case, the user action can be a called phone number, and the processor actions can be listed phone numbers. The processing device can be operated in a single user environment, multiple user environment, or both. The processor action can be selected in response to the received user action or in response to some other information or event.
[0068] In any event, the processor action selection is based on an action probability distribution that contains a plurality of probability values corresponding to the plurality of processor actions. For example, the selected processor action can correspond to the highest probability value within the action probability distribution, or can correspond to a pseudo-random selection of a value within the action probability distribution. The action probability distribution may be initially generated with equal probability values (e.g., if it is not desired that the processing device learn more quickly of if no assumptions are made as to which processor actions are more likely to be selected in the near future) or unequal probability values (if it is desired that the processing device learn more quickly, and if it is assumed that there are certain processor actions that are more likely to be selected in the near future). Preferably, the action probability distribution is normalized.
[0069] The method further comprises determining an outcome of one or both of the received user action and selected processor action. By way of non-limiting example, the outcome can be represented by one of only two values, e.g., zero (outcome is not successful) and one (outcome is successful), one of a finite range of real numbers (higher numbers mean outcome is more successful), or one of a range of continuous values (the higher the number, the more successful the outcome is). If the outcome is based thereon, the selected processor action can be a currently selected processor action, previously selected processor action (lag learning), or subsequently selected processor action (lead learning).
[0070] The method further comprises updating the action probability distribution based on the outcome. A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device. The action probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0071] Lastly, the method comprises repeating the foregoing steps, wherein the action probability distribution is prevented from substantially converging to a single probability value. It is worth noting that absent this step, a single best action or a group of best actions for a given predetermined environment will be determined. In the case of a changing environment, however, this may ultimately diverge from the objectives to be achieved. Thus, a single best action is not assumed over a period of time, but rather assumes that there is a dynamic best action that changes over the time period. Because the action probability value for any best action will not be unity, selection of the best action at any given time is not ensured, but will merely tend to occur, as dictated by its corresponding probability value. Thus, it is ensured that the objective(s) to be met are achieved over time.
[0072] In accordance with a sixteenth aspect of the present inventions, a processing device comprises a probabilistic learning module having a learning automaton configured for learning a plurality of processor actions in response to a plurality of actions performed by a user, and an intuition module configured for preventing the probabilistic learning module from substantially converging to a single processor action. The processing device can be operated in a single user, multiple user environment, or both. The intuition module can be, e.g., deterministic, quasi-deterministic, or probabilistic. It can use, e.g., artificial intelligence, expert systems, neural networks, or fuzzy logic.
[0073] In the preferred embodiment, the probabilistic learning module may include an action selection module configured for selecting one of a plurality of processor actions. The action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The probabilistic learning module may further comprise an outcome evaluation module configured for determining an outcome of either or both of the received user action and selected processor action. The probabilistic learning module may further comprise a probability update module configured for updating the action probability distribution based on the outcome.
[0074] In accordance with a seventeenth aspect of the present inventions, a method of providing learning capability to a processing device having a functionality independent of determining an optimum action. The method comprises receiving an action performed by a user, and selecting one of a plurality of processor actions that affects the functionality of the electronic device. By way of non-limiting example, the processing device can be a computer game, in which case, the user action can be a player move, and the processor actions can be game moves. Or the processing device can be an educational toy, in which case, the user action can be a child action, and the processor actions and be toy actions. Or the processing device can be a telephone system, in which case, the user action can be a called phone number, and the processor actions can be listed phone numbers. Or the processing device can be a television channel control system, in which case, the user action can be a watched television channel, and the processor actions can be listed television channels. The processing device can be operated in a single user environment, multiple user environment, or both. The processor action can be selected in response to the received user action or in response to some other information or event.
[0075] In any event, the processor action selection is based on an action probability distribution that contains a plurality of probability values corresponding to the plurality of processor actions. For example, the selected processor action can correspond to the highest probability value within the action probability distribution, or can correspond to a pseudo-random selection of a value within the action probability distribution. The action probability distribution may be initially generated with equal probability values (e.g., if it is not desired that the processing device learn more quickly of if no assumptions are made as to which processor actions are more likely to be selected in the near future) or unequal probability values (if it is desired that the processing device learn more quickly, and if it is assumed that there are certain processor actions that are more likely to be selected in the near future). Preferably, the action probability distribution is normalized.
[0076] The method further comprises determining an outcome of one or both of the received user action and selected processor action. By way of non-limiting example, the outcome can be represented by one of only two values, e.g., zero (outcome is not successful) and one (outcome is successful), one of a finite range of real numbers (higher numbers mean outcome is more successful), or one of a range of continuous values (the higher the number, the more successful the outcome is). If the outcome is based thereon, the selected processor action can be a currently selected processor action, previously selected processor action (lag learning), or subsequently selected processor action (lead learning).
[0077] The method further comprises updating the action probability distribution based on the outcome. A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device. The action probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0078] In accordance with an eighteenth aspect of the present inventions, a processing device (such as, e.g., a computer game, educational toy, telephone system, television channel control system, etc.) comprises an action selection module configured for selecting one of a plurality of processor actions, wherein the selected processor action affects the processing device function. The action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The processing device further comprises an outcome evaluation module configured for determining an outcome of either or both of the received user action and selected processor action. The probabilistic learning module further comprises a probability update module configured for updating the action probability distribution based on the outcome. The processing device can be operated in a single user, multiple user environment, or both. The intuition module can be, e.g., deterministic, quasi-deterministic, or probabilistic. It can use, e.g., artificial intelligence, expert systems, neural networks, or fuzzy logic.
[0079] In accordance with a nineteenth aspect of the present inventions, a method of providing learning capability to a processing device having one or more objectives, comprises receiving actions from a plurality of users, and selecting one or more processor actions from a plurality of processor actions. By way of non-limiting example, the processing device can be a computer game, in which case, the user action can be a player move, and the processor actions can be game moves. Or the processing device can be an educational toy, in which case, the user action can be a child action, and the processor actions and be toy actions. Or the processing device can be a telephone system, in which case, the user action can be a called phone number, and the processor actions can be listed phone numbers. Or the processing device can be a television channel control system, in which case, the user action can be a watched television channel, and the processor actions can be listed television channels. The one or more processor actions can be a single processor action or multiple processor actions corresponding to the plurality of user actions. The processor action(s) selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The processor action(s) can be selected in response to the received user actions or in response to some other information or event.
[0080] The method further comprises determining one or more outcomes based on one or both of the plurality of user actions and the selected processor action(s). The one or more outcomes can be, e.g., a single outcome that corresponds to the plurality of user actions or plurality of outcomes that respectively corresponds to the plurality of user actions. Optionally, the outcome(s) are only determined after several iterations of the user action receiving and processor action selection, e.g., to save processing power.
[0081] The method further comprises updating the action probability distribution based on the outcome(s). The action probability distribution can be updated when a predetermined period of time has expired or otherwise when some condition has been satisfied, e.g., to synchronize user actions that are asynchronously received, or it can be updated in response to the receipt of each user. Optionally, the action probability distribution is only updated after several iterations of the user action receiving, processor action selection, and outcome determination, e.g., to save processing power.
[0082] A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device.
[0083] The method further comprises modifying one or more of the processor action selection, outcome determination, and action probability distribution update steps based on the objective(s). These steps can be modified in any combination of a variety of ways. For example, one of a predetermined plurality of algorithms employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be selected, or one or more parameters within an algorithm employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be modified.
[0084] Optionally, the method may further comprise determining one or more performance indexes indicative of a performance of the processing device relative to the objective(s) of the processing device, wherein the modification is based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0085] In accordance with a twentieth aspect of the present inventions, a method of providing learning capability to a processing device having one or more objectives, comprises receiving actions from users divided amongst a plurality of user sets. Each user set may have a single user or multiple users.
[0086] For each user set, the method further comprises (1) selecting one or more of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions; (2) determining one or more outcomes based on one or more actions from each user set and selected processor action(s); (3) updating the action probability distribution using a learning automaton based on the outcome(s); and (4) modifying one or more of the processor action selection, outcome determination, and action probability distribution update steps based on the one or more objective(s). The steps can be implemented in any variety of ways, as previously discussed above.
[0087] In accordance with a twenty-first aspect of the present inventions, a processing device comprises a probabilistic learning module having a learning automaton configured for learning a plurality of processor actions in response to a plurality of actions performed by a user, and an intuition module configured for modifying a functionality of the probabilistic learning module based on one or more objectives of the processing device, e.g., by selecting one of a plurality of algorithms used by the learning module, or modifying a parameter of an algorithm employed by the learning module.
[0088] Optionally, the intuition module can be further configured for determining one or more performance indexes indicative of a performance of the probabilistic learning module relative to the objective(s), and for modifying the probabilistic learning module functionality based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0089] In the preferred embodiment, the probabilistic learning module may include one or more action selection modules configured for selecting one or more of a plurality of processor actions. The one or more selected processor actions can be a single processor action or multiple processor actions corresponding to the plurality of user actions. The action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The probabilistic learning module may further comprise one or more outcome evaluation modules configured for determining one or more outcomes based on one or both of the plurality of user actions and the selected processor actions(s). The one or more outcomes can be, e.g., a single outcome that corresponds to the plurality of user actions or plurality of outcomes that respectively corresponds to the plurality of user actions. The probabilistic learning module may further comprise a probability update module configured for updating the action probability distribution based on the outcome(s). When modifying the functionality of the learning module, the intuition module may modify a functionality of any combination of the action selection module, outcome evaluation module, and probability update module.
[0090] Optionally, the processing device can comprise a server, a plurality of computers, and a network. Depending on how the processing capability is to be distributed between the server and computers, any combination of the action selection module(s), outcome evaluation module(s), and probability update module can be contained within the server and computers.
[0091] For example, virtually all of the processing capability can be within the server, in which case, the server can contain the action selection module(s), and outcome evaluation module(s), and probability update module. The plurality of computers will then merely be configured for respectively generated the plurality of user actions. The network will then be configured for transmitting the plurality of user actions from the plurality of computers to the server and for transmitting the selected processor action(s) from the server to the plurality of computers.
[0092] If the one or more action selection modules comprises a plurality of action selection modules for selecting a plurality of processor actions, some of the processing capability can be offloaded to the computers. In this case, the server can contain the outcome evaluation module(s) and probability update module. The plurality of computers will then contain the action selection modules. The network will then be configured for transmitting the plurality of user actions and selected plurality of processor actions from the plurality of computers to the server. If the one or more outcome evaluation modules comprises a plurality of outcome evaluation modules for determining a plurality of outcomes, even more of the processing capability can be offloaded to the computers. In this case, the server can merely contain the probability update module, and the plurality of computers can contain the action selection modules and outcome evaluation modules. The network will then be configured for transmitting the plurality of outcomes from the plurality of computers to the server.
[0093] If the plurality of users are divided amongst a plurality of user sets, the probabilistic learning module can comprises for each user set one or more action selection modules, one or more outcome evaluation modules, and one or more probability update modules. Each user set may have a single user or multiple users. The functionality of these modules can be implemented in any variety of ways, as previously discussed above. The processing capability of these modules can be distributed between a server and a plurality of computers, as previously discussed above.
[0094] In accordance with a twenty-second aspect of the present inventions, a method of providing learning capability to a processing device having one or more objectives, comprises receiving a plurality of user actions, and selecting one or more processor actions from a plurality of processor actions. By way of non-limiting example, the processing device can be, e.g., a computer game, in which case, the user actions can be player moves, and the processor actions can be game moves. The user actions can be received from a single user or multiple users. The one or more processor actions can be a single processor action or multiple processor actions corresponding to the plurality of user actions. The processor action(s) selection is based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The processor action(s) can be selected in response to the received user actions or in response to some other information or event.
[0095] The method further comprises weighting the user actions. In this manner, each of the user actions affects the learning process differently. For example, if the user actions were received from a plurality of users, the weightings can be based on a skill level of the users. Thus, the effect that each user has on the learning process will be based on the skill level of that user.
[0096] The method further comprises determining one or more outcomes based on the plurality of weighted user actions. The one or more outcomes can be, e.g., a single outcome that corresponds to the plurality of user actions or plurality of outcomes that respectively corresponds to the plurality of user actions. Optionally, the outcome(s) are only determined after several iterations of the user action receiving and processor action selection, e.g., to save processing power.
[0097] The method further comprises updating the action probability distribution. The action probability distribution can be updated when a predetermined period of time has expired or otherwise when some condition has been satisfied, e.g., to synchronize user actions that are asynchronously received, or it can be updated in response to the receipt of each user. Optionally, the action probability distribution is only updated after several iterations of the user action receiving, processor action selection, and outcome determination, e.g., to save processing power.
[0098] A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device. The action probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0099] Optionally, the method may further comprise modifying one or more of the processor action selection, outcome determination, and action probability distribution update steps based on the objective(s). These steps can be modified in any combination of a variety of ways. For example, one of a predetermined plurality of algorithms employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be selected, or one or more parameters within an algorithm employed by the processor action selection, outcome determination, and action probability distribution update steps can be modified. The outcome determination modification can comprise modifying a weighting of the user actions.
[0100] Optionally, the method may further comprise determining one or more performance indexes indicative of a performance of the processing device relative to the objective(s) of the processing device, wherein the modification is based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0101] In accordance with a twenty-third aspect of the present inventions, a processing device comprises an action selection module configured for selecting one or more of a plurality of processor actions. The action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The one or more selected processor actions can be a single processor action or multiple processor actions corresponding to the plurality of user actions. The processing device further comprises an outcome evaluation module configured for weighting a plurality of received user actions, and for determining one or more outcomes based on the plurality of weighted user actions. The user actions can be received from a single user or multiple users. The one or more outcomes can be, e.g., a single outcome that corresponds to the plurality of user actions or plurality of outcomes that respectively corresponds to the plurality of user actions. The processing device further comprises a probability update module configured for updating the action probability distribution based on the outcome(s). The probability update module may optionally include a learning automaton to update the action probability distribution.
[0102] The processing device may optionally include an intuition module configured for modifying a functionality of any combination of the action selection module, outcome evaluation module, and probability update module based on one or more objectives of the processing device. The intuition module can be further configured for determining one or more performance indexes indicative of a performance of the probabilistic learning module relative to the objective(s), and for modifying the probabilistic learning module functionality based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0103] In accordance with a twenty-fourth aspect of the present inventions, a method of providing learning capability to a processing device having one or more objectives, comprises receiving a plurality of user actions, and selecting one or more processor actions from a plurality of processor actions. By way of non-limiting example, the processing device can be, e.g., a computer game, in which case, the user actions can be player moves, and the processor actions can be game moves. The user actions can be received from a single user or multiple users. The processor action selection is based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The processor action can be selected in response to the received user actions or in response to some other information or event.
[0104] The method further comprises determining a success ratio of a selected processor action relative to the plurality of user actions, and comparing the determined success ratio to a reference success ratio (e.g., simple majority, minority, super majority, unanimity, equality). The method further comprises determining an outcome of the success ratio comparison. For example, if the reference success ratio for the selected processor action is a majority, and there are three user actions received, the outcome may equal “1” (indicating a success) if the selected processor action is successful relative to two or more of the three user actions, and may equal “0” (indicating a failure) if the selected processor action is successful relative to one or none of the three user actions.
[0105] The method further comprises updating the action probability distribution. The action probability distribution can be updated when a predetermined period of time has expired or otherwise when some condition has been satisfied, e.g., to synchronize user actions that are asynchronously received, or it can be updated in response to the receipt of each user. Optionally, the action probability distribution is only updated after several iterations of the user action receiving, processor action selection, and outcome determination, e.g., to save processing power.
[0106] A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device. The action probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0107] Optionally, the method may further comprise modifying one or more of the processor action selection, outcome determination, and action probability distribution update steps based on the objective(s). These steps can be modified in any combination of a variety of ways. For example, one of a predetermined plurality of algorithms employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be selected, or one or more parameters within an algorithm employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be modified. The outcome determination modification can comprise modifying the reference success ratio.
[0108] Optionally, the method may further comprise determining one or more performance indexes indicative of a performance of the processing device relative to the objective(s) of the processing device, wherein the modification is based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0109] In accordance with a twenty-fifth aspect of the present inventions, a processing device comprises an action selection module configured for selecting one of a plurality of processor actions. The action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The processing device further comprises an outcome evaluation module configured for determining a success ratio of the selected processor action relative to a plurality of user actions, for comparing the determined success ratio to a reference success ratio, and for determining an outcome of the success ratio comparison. The user actions can be received from a single user or multiple users. The processing device further comprises a probability update module configured for updating the action probability distribution based on the outcome. The probability update module may optionally include a learning automaton to update the action probability distribution.
[0110] The processing device may optionally include an intuition module configured for modifying a functionality of any combination of the action selection module, outcome evaluation module, and probability update module based on one or more objectives of the processing device. The intuition module can be further configured for determining one or more performance indexes indicative of a performance of the probabilistic learning module relative to the objective(s), and for modifying the probabilistic learning module functionality based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0111] In accordance with a twenty-sixth aspect of the present inventions, a method of providing learning capability to a processing device having one or more objectives, comprises receiving actions from a plurality of users, and selecting one of a plurality of processor actions. By way of non-limiting example, the processing device can be, e.g., a computer game, in which case, the user actions can be player moves, and the processor actions can be game moves. The processor action selection is based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The processor action can be selected in response to the received user actions or in response to some other information or event.
[0112] The method further comprises determining if the selected processor action has a relative success level (e.g., a greatest, least, or average success level) for a majority of the plurality of users. The relative success level can be determined in a variety of ways. For example, separate action probability distributions for the plurality of users can be maintained, and then the relative success level of the selected processor action can be determined from the separate action probability distributions. As another example, an estimator success table for the plurality of users can be maintained, and then the relative success level of the selected processor action can be determined from the estimator table.
[0113] The method further comprises determining an outcome of the success ratio comparison. For example, if the relative success level is the greatest success level, the outcome may equal “1” (indicating a success) if the selected processor action is the most successful for the maximum number of users, and may equal “0” (indicating a failure) if the selected processor action is not the most successful for the maximum number of users.
[0114] The method further comprises updating the action probability distribution. The action probability distribution can be updated when a predetermined period of time has expired or otherwise when some condition has been satisfied, e.g., to synchronize user actions that are asynchronously received, or it can be updated in response to the receipt of each user. Optionally, the action probability distribution is only updated after several iterations of the user action receiving, processor action selection, and outcome determination, e.g., to save processing power.
[0115] A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device. The action probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0116] Optionally, the method may further comprise modifying one or more of the processor action selection, outcome determination, and action probability distribution update steps based on the objective(s). These steps can be modified in any combination of a variety of ways. For example, one of a predetermined plurality of algorithms employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be selected, or one or more parameters within an algorithm employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be modified. The outcome determination modification can comprise modifying the relative success level.
[0117] Optionally, the method may further comprise determining one or more performance indexes indicative of a performance of the processing device relative to the objective(s) of the processing device, wherein the modification is based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0118] In accordance with a twenty-seventh aspect of the present inventions, a processing device comprises an action selection module configured for selecting one of a plurality of processor actions. The action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor actions. The processing device further comprises an outcome evaluation module configured for determining if the selected processor action has a relative success level for a majority of a plurality of users, and for determining an outcome of the success determination. The processing device further comprises a probability update module configured for updating the action probability distribution based on the outcome. The probability update module may optionally include a learning automaton to update the action probability distribution.
[0119] The processing device may optionally include an intuition module configured for modifying a functionality of any combination of the action selection module, outcome evaluation module, and probability update module based on one or more objectives of the processing device. The intuition module can be further configured for determining one or more performance indexes indicative of a performance of the probabilistic learning module relative to the objective(s), and for modifying the probabilistic learning module functionality based on the performance index(es). The one or more performance indexes can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0120] In accordance with a twenty-eighth aspect of the present inventions, a method of providing learning capability to a processing device having one or more objectives, comprises selecting one or more processor actions from a plurality of processor actions that are linked to one or more pluralities of user parameters (such as, e.g., users and/or user actions) to generate action pairs, or trios or higher numbered groupings.
[0121] By way of non-limiting example, the processing device can be, e.g., a computer game, in which case, the user actions can be player moves, and the processor actions can be game moves. The user action(s) can be a single user action received from a user or multiple user actions received from a single or multiple users. The one or more processor actions can be a single processor action or multiple processor actions corresponding to the plurality of user actions. The processor action(s) selection is based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of linked processor actions. The method may further comprise receiving one or more user actions. The user action(s) can be a single user action received from a user or multiple user actions received from a single or multiple users. The processor action(s) can be selected in response to the received user action(s) or in response to some other information or event.
[0122] The method further comprises linking the selected process action(s) with one or more of the plurality of user parameter, and determining one or more outcomes based on the plurality of selected linked processor action(s). The one or more outcomes can be, e.g., a single outcome that corresponds to the plurality of user actions or plurality of outcomes that respectively corresponds to the plurality of user actions. Optionally, the outcome(s) are only determined after several iterations of the user action receiving and processor action selection, e.g., to save processing power.
[0123] The method further comprises updating the action probability distribution. The action probability distribution can be updated when a predetermined period of time has expired or otherwise when some condition has been satisfied, e.g., to synchronize user actions that are asynchronously received, or it can be updated in response to the receipt of each user. Optionally, the action probability distribution is only updated after several iterations of the user action receiving, processor action selection, and outcome determination, e.g., to save processing power.
[0124] A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor actions and/or user actions, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device. The action probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0125] Optionally, the method may further comprise modifying one or more of the processor action selection, outcome determination, and action probability distribution update steps. These steps can be modified in any combination of a variety of ways. For example, one of a predetermined plurality of algorithms employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be selected, or one or more parameters within an algorithm employed by the processor action selection, outcome determination, and action probability distribution update step(s) can be modified. The outcome determination modification can comprise modifying a weighting of the user actions.
[0126] Optionally, the method may further comprise determining one or more performance indexes indicative of a performance of the processing device relative to the objective(s) of the processing device, wherein the modification is based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0127] In accordance with a twenty-ninth aspect of the present inventions, a processing device comprises an action selection module configured for selecting one or more of a plurality of processor actions that are respectively linked to a plurality of user parameters. The action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of linked processor actions. The one or more selected processor actions can be a single processor action or multiple processor actions corresponding to the plurality of user actions. The processing device further comprises an outcome evaluation module configured for linking the selected process action(s) with one or more of the plurality of user parameters, and for determining one or more outcomes based on the one or more linked processor actions.
[0128] The action selection module can be configured for receiving one or more user actions. The user actions can be received from a single user or multiple users. The one or more outcomes can be, e.g., a single outcome that corresponds to the plurality of user actions or plurality of outcomes that respectively corresponds to the plurality of user actions. The processing device further comprises a probability update module configured for updating the action probability distribution based on the outcome(s). The probability update module may optionally include a learning automaton to update the action probability distribution.
[0129] The processing device may optionally include an intuition module configured for modifying a functionality of any combination of the action selection module, outcome evaluation module, and probability update module based on one or more objectives of the processing device. The intuition module can be further configured for determining one or more performance indexes indicative of a performance of the probabilistic learning module relative to the objective(s), and for modifying the probabilistic learning module functionality based on the performance index(es). The one or more performance index(es) can be a single index that corresponds to the plurality of user actions or a plurality of performance indexes that respectively correspond to the plurality of user actions.
[0130] In accordance with a thirtieth aspect of the present inventions, a method of providing learning capability to a processing device (e.g., a telephone or a television channel control system) having an objective (e.g., anticipating called phone numbers or watched television channels), comprises generating a list containing a plurality of listed items with an associated item probability distribution, which comprises a plurality of probability values corresponding to the plurality of listed items. The listed items can be, e.g., telephone numbers or television channels. Preferably, the item probability distribution is normalized.
[0131] The method further comprises selecting one or more items from the plurality of listed items based on the item probability distribution. In the preferred method, the selected item(s) corresponds to the highest probability values in the item probability distribution, and are placed in an order according to the corresponding probability values. In this manner, the “favorite” item(s) can be communicated to the user.
[0132] The method further comprises determining a performance index indicative of a performance of the processing device relative to its objective. For example, the method may comprise identifying an item associated with an action, and determining if the identified item matches any listed items contained in the list and/or selected item(s). In this case, the performance index will be derived from this determination. The performance index may be instantaneous, e.g., if a currently identified item is used, or cumulative, e.g., if a tracked percentage of identified items is used.
[0133] The method further comprises modifying the item probability distribution based on the performance index. The item probability distribution can be modified in a variety of ways. For example, the item probability distribution can be modified by updating the item probability distribution, e.g., using a reward-inaction update. Or the item probability distribution can be modified by increasing a probability value corresponding to a particular listed item, or adding a probability value, e.g., when a new item is added to the list. In this case, the probability value can be replaced with the added probability value to, e.g., to minimize storage space.
[0134] In one preferred method, the item probability distribution is modified by updating it if the identified item matches any listed item. For example, the item probability distribution update can comprise a reward-inaction update, e.g., by rewarding the corresponding probability value. The method may further comprises adding a listed item corresponding to the identified item to the item list if the identified item does not match any listed item. In this case, the item probability distribution will be modified by adding a probability value corresponding to the added listed item to the item probability distribution. Another item on the item list can be replaced with the added listed item, and another probability value corresponding to the replaced listed item can be replaced with the added probability value.
[0135] In another preferred method, the item probability distribution is modified by updating it only if the identified item matches an item within the selected item(s). For example, the item probability distribution update can comprise a reward-inaction update, e.g., by rewarding the corresponding probability value. This preferred method may further comprise modifying the item probability distribution by increasing a corresponding probability value if the identified item matches a listed item that does not correspond to an item within the selected items. The method may further comprise adding a listed item corresponding to the identified item to the item list if the identified item does not match any listed item. In this case, the item probability distribution will be modified by adding a probability value corresponding to the added listed item to the item probability distribution. Another item on the item list can be replaced with the added listed item, and another probability value corresponding to the replaced listed item can be replaced with the added probability value.
[0136] The item probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next item probability distribution is a function of the current item probability distribution. Advantageously, updating of the item probability distribution using a learning automaton is based on a frequency of the items, as well as the time ordering of these items. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the processing device. Alternatively, the item probability distribution can be purely frequency-based. For example, item probability distribution can be based on a moving average.
[0137] The method may optionally comprise generating another item list containing at least another plurality of listed items and an item probability distribution comprising a plurality of probability values corresponding to the other plurality of listed items. This optional method further comprises selecting another set of items from the other plurality of items based on the other item probability distribution. An item associated with an action can then be identified, in which case, the method further comprises determining if the identified item matches any listed item contained in the item list. Another item associated with another action can also be identified, in which case, the method further comprises determining if the other identified item matches any listed item contained in the other item list. The performance index is this case will be derived from these matching determinations.
[0138] The two item lists can be used to distinguish between days of the week or time of day. For example, the method may further comprise identifying an item associated with an action, determining the current day of the week, selecting one of the two item lists based on the current day determination, and determining if the identified item matches any listed item contained in the selected item list. Or the method may further comprise identifying an item associated with another action, determining the current time of the day, selecting one of the two item lists based on the current time determination, and determining if the identified item matches any listed item contained in the selected item list.
[0139] In accordance with a thirty-first aspect of the present inventions, a processing device (e.g., a telephone or television channel control system) having an objective (e.g., anticipating called phone number or watched television channels) comprises a probabilistic learning module configured for learning favorite items of a user in response to identified user actions, and an intuition module configured for modifying a functionality of the probabilistic learning module based on the objective. The probabilistic learning module can include a learning automaton or can be purely frequency-based. The learning module and intuition module can be self-contained in a single device or distributed within several devices. For example, in the case of a phone system, the learning module and intuition module can be contained with the phone, a server or both. In the case of a television channel control system, the learning module and intuition module can be contained with a remote control, a cable box, video cassette recorder, television, or any combination thereof.
[0140] Optionally, the probabilistic learning module is configured for determining a performance index indicative of a performance of the probabilistic learning module relative to the objective, and the intuition module is configured for modifying the probabilistic learning module functionality based on the performance index. As previously described, the performance index can be instantaneous or cumulative.
[0141] In the preferred embodiment, the probabilistic learning module comprises an item selection module configured for selecting the favorite item(s) from a plurality of items based on an item probability distribution comprising a plurality of probability values corresponding to the plurality of listed items. The favorite items can correspond to the highest probability values in the item probability distribution. The item selection module can be further configured for placing the favorite numbers in an order according to corresponding probability values. The probabilistic learning module further comprises an outcome evaluation module configured for determining if identified items match any listed item contained in the item list, and a probability update module, wherein the intuition module is configured for modifying the probability update module based on the matching determinations.
[0142] The intuition module can modify the probability update module in a variety of ways. For example, the intuition module can be configured for modifying the probability update module by directing it to update the item probability distribution if any of the identified items matches any listed item. For example, a reward-inaction update can be used, e.g., by rewarding the corresponding probability value. The intuition module can further be configured for modifying the probability update module by adding a listed item corresponding to the identified item to the item list and adding a probability value corresponding to the added listed item to the item probability distribution if the identified item does not match any listed item. In this case, another item on the item list may be replaced with the added listed item, and another probability value corresponding to the replaced listed item can be replaced with the added probability value.
[0143] As another example, the intuition module can be configured for modifying the probability update module by directing it to update the item probability distribution only if the identified plurality of items matches a listed item corresponding to one of the favorite items. For example, a reward-inaction update can be used, e.g., by rewarding the corresponding probability value. The intuition module can further be configured for modifying the probability update module by increasing a corresponding probability value if the identified item matches a listed item that does not correspond to one of the favorite items. The intuition module can further be configured for modifying the probability update module by adding a listed item corresponding to the identified item to the item list and adding a probability value corresponding to the added listed item to the item probability distribution if the identified item does not match any listed item. In this case, another item on the item list may be replaced with the added listed item, and another probability value corresponding to the replaced listed item can be replaced with the added probability value.
[0144] In an optional embodiment, the favorite items can be divided into first and second favorite item lists, in which case, the probabilistic learning module can be configured for learning the first favorite item list in response to the identification of item associated items during a first time period, and for learning the second favorite item list in response to item associated items during a second time period. For example, the first time period can include weekdays, and the second time period can include weekends. Or the first time period can include days, and the second time period can include evenings.
[0145] In accordance with a thirty-second aspect of the present inventions, a method of providing learning capability to a processing device (e.g., a television channel control system) having an objective (e.g., anticipating watched television channels), comprises generating a plurality of lists respectively corresponding to a plurality of item parameter values (e.g., television channel parameters). Each of the plurality of lists contains a plurality of listed items with an associated item probability distribution comprising a plurality of probability values corresponding to the plurality of listed items.
[0146] The method further comprises selecting a list corresponding to a parameter value exhibited by a currently identified item (e.g., a currently watched television channel), and in the selected list, selecting one or more listed items from the plurality of listed items based on the item probability distribution. The method further comprises determining a performance index indicative of a performance of the processing device relative to its objective. For example, the method may comprise identifying an action associated item exhibiting a parameter value, selecting a list corresponding to the identified parameter value, and determining if the identified item matches any listed items contained in the selected list. In this case, the performance index will be derived from this determination. The performance index may be instantaneous, e.g., if a currently identified item is used, or cumulative, e.g., if a tracked percentage of identified items is used. The item probability distribution can be modified in a variety of ways, including those described above.
[0147] By way of non-limiting example, the use of a plurality of lists with respective associated parameter values allows an objective of the processing device (e.g., anticipating the favorite items of the user) to be better achieved by focussing on the list that more closely matches the item selection pattern that is currently exhibiting the corresponding parameter value.
[0148] In accordance with a thirty-third aspect of the present inventions, a method of providing learning capability to a phone number calling system, such as, e.g., a mobile phone, having an objective of anticipating called phone numbers, comprises generating a phone list containing at least a plurality of listed phone numbers and a phone number probability distribution comprising a plurality of probability values corresponding to the plurality of listed phone numbers. The plurality of probability values can correspond to all phone numbers within the phone list or only the plurality of phone numbers. Preferably, the phone number probability distribution is normalized.
[0149] The method further comprises selecting a set of phone numbers from the plurality of listed phone numbers based on the phone number probability distribution. In the preferred method, the selected set of phone numbers are communicated to a user of the phone number calling system, e.g., by displaying it. The phone number set can be a single phone number, but preferably is a plurality of phone numbers from which the user can select. In this case, the user will be able to select the phone number from the phone number set to make a phone call. In addition, the selected phone number set corresponds to the highest probability values in the phone number probability distribution, and are placed in an order according to the corresponding probability values. In this manner, the “favorite” phone numbers will be communicated to the user.
[0150] The method further comprises determining a performance index indicative of a performance of the phone number calling system relative to the objective of anticipating called phone numbers. For example, the method may comprises identifying a phone number associated with a phone call, and determining if the identified phone number matches any listed phone number contained in the phone number list and/or selected phone number(s). In this case, the performance index will be derived from this determination. The identified phone number can be, e.g., associated with an outgoing phone call or an incoming phone call. The performance index may be instantaneous, e.g., if a currently identified phone number is used, or cumulative, e.g., if a tracked percentage of identified phone numbers is used.
[0151] The method further comprises modifying the phone number probability distribution based on the performance index. The phone number probability distribution can be modified in a variety of ways. For example, the phone number probability distribution can be modified by updating the phone number probability distribution, e.g., using a reward-inaction update. Or the phone number probability distribution can be modified by increasing a probability value corresponding to a particular listed phone number, or adding a probability value, e.g., when a new phone number is added to the list. In this case, the probability value can be replaced with the added probability value to, e.g., to minimize storage space.
[0152] In one preferred method, the phone number probability distribution is modified by updating it if the identified phone number matches any listed phone number. For example, the phone number probability distribution update can comprise a reward-inaction update, e.g., by rewarding the corresponding probability value. The method may further comprises adding a listed phone number corresponding to the identified phone number to the phone list if the identified phone number does not match any listed phone number. In this case, the phone number probability distribution will be modified by adding a probability value corresponding to the added listed phone number to the phone number probability distribution. Another phone number on the phone list can be replaced with the added listed phone number, and another probability value corresponding to the replaced listed phone number can be replaced with the added probability value.
[0153] In another preferred method, the phone number probability distribution is modified by updating it only if the identified phone number matches a phone number within the selected phone number set. For example, the phone number probability distribution update can comprise a reward-inaction update, e.g., by rewarding the corresponding probability value. This preferred method may further comprise modifying the phone number probability distribution by increasing a corresponding probability value if the identified phone number matches a listed phone number that does not correspond to a phone number within the selected phone number set. The method may further comprise adding a listed phone number corresponding to the identified phone number to the phone list if the identified phone number does not match any listed phone number. In this case, the phone number probability distribution will be modified by adding a probability value corresponding to the added listed phone number to the phone number probability distribution. Another phone number on the phone list can be replaced with the added listed phone number, and another probability value corresponding to the replaced listed phone number can be replaced with the added probability value.
[0154] The phone number probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next phone number probability distribution is a function of the current phone number probability distribution. Advantageously, updating of the phone number probability distribution using a learning automaton is based on a frequency of the phone numbers, as well as the time ordering of these phone numbers. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the phone system. Alternatively, the phone number probability distribution can be purely frequency-based. For example, phone number probability distribution can be based on a moving average.
[0155] The method may optionally comprise generating another phone list containing at least another plurality of listed phone numbers and a phone number probability distribution comprising a plurality of probability values corresponding to the other plurality of listed phone numbers. This optional method further comprises selecting another set of phone numbers from the other plurality of phone numbers based on the other phone number probability distribution. A phone number associated with a phone call can then be identified, in which case, the method further comprises determining if the identified phone number matches any listed phone number contained in the phone number list. Another phone number associated with a phone call can also be identified, in which case, the method further comprises determining if the other identified phone number matches any listed phone number contained in the other phone number list. The performance index is this case will be derived from these matching determinations.
[0156] The two phone lists can be used to distinguish between days of the week or time of day. For example, the method may further comprise identifying a phone number associated with a phone call, determining the current day of the week, selecting one of the two phone lists based on the current day determination, and determining if the identified phone number matches any listed phone number contained in the selected phone number list. Or the method may further comprise identifying a phone number associated with a phone call, determining the current time of the day, selecting one of the two phone lists based on the current time determination, and determining if the identified phone number matches any listed phone number contained in the selected phone number list.
[0157] In accordance with a thirty-fourth aspect of the present inventions, a phone number calling system having an objective of anticipating called phone numbers, comprises a probabilistic learning module configured for learning favorite phone numbers of a user in response to phone calls, and an intuition module configured for modifying a functionality of the probabilistic learning module based on the objective of anticipating called phone numbers. The phone calls may be, e.g., incoming and/or outgoing phone calls. The probabilistic learning module can include a learning automaton or can be purely frequency-based. The learning module and intuition module can be self-contained in a single device, e.g., a telephone or a server, or distributed within several devices, e.g., both the server and phone.
[0158] Optionally, the probabilistic learning module is configured for determining a performance index indicative of a performance of the probabilistic learning module relative to the objective of anticipating called phone numbers, and the intuition module is configured for modifying the probabilistic learning module functionality based on the performance index. As previously described, the performance index can be instantaneous or cumulative.
[0159] Optionally, the phone number calling system comprises a display for displaying the favorite phone numbers. The phone number calling system may further comprise one or more selection buttons configured for selecting one of the favorite phone numbers to make a phone call.
[0160] In the preferred embodiment, the probabilistic learning module comprises a phone number selection module configured for selecting the favorite phone numbers from a plurality of phone numbers based on a phone number probability distribution comprising a plurality of probability values corresponding to the plurality of listed phone numbers. The plurality of probability values can correspond to all phone numbers within the phone list or only the plurality of phone numbers. The probabilistic learning module further comprises an outcome evaluation module configured for determining if identified phone numbers associated with the phone calls match any listed phone number contained in the phone number list, and a probability update module, wherein the intuition module is configured for modifying the probability update module based on the matching determinations. In this case, the favorite phone numbers can correspond to the highest probability values in the phone number probability distribution. The phone number selection module can be further configured for placing the favorite numbers in an order according to corresponding probability values.
[0161] The intuition module can modify the probability update module in a variety of ways. For example, the intuition module can be configured for modifying the probability update module by directing it to update the phone number probability distribution if any of the identified phone numbers matches any listed phone number. For example, a reward-inaction update can be used, e.g., by rewarding the corresponding probability value. The intuition module can further be configured for modifying the probability update module by adding a listed phone number corresponding to the identified phone number to the phone list and adding a probability value corresponding to the added listed phone number to the phone number probability distribution if the identified phone number does not match any listed phone number. In this case, another phone number on the phone list may be replaced with the added listed phone number, and another probability value corresponding to the replaced listed phone number can be replaced with the added probability value.
[0162] As another example, the intuition module can be configured for modifying the probability update module by directing it to update the phone number probability distribution only if the identified plurality of phone numbers matches a listed phone number corresponding to one of the favorite phone numbers. For example, a reward-inaction update can be used, e.g., by rewarding the corresponding probability value. The intuition module can further be configured for modifying the probability update module by increasing a corresponding probability value if the identified phone number matches a listed phone number that does not correspond to one of the favorite phone numbers. The intuition module can further be configured for modifying the probability update module by adding a listed phone number corresponding to the identified phone number to the phone list and adding a probability value corresponding to the added listed phone number to the phone number probability distribution if the identified phone number does not match any listed phone number. In this case, another phone number on the phone list may be replaced with the added listed phone number, and another probability value corresponding to the replaced listed phone number can be replaced with the added probability value.
[0163] In an optional embodiment, the favorite phone numbers can be divided into first and second favorite phone number lists, in which case, the probabilistic learning module can be configured for learning the first favorite phone number list in response to phone calls during a first time period, and for learning the second favorite phone number list in response to phone calls during a second time period. For example, the first time period can include weekdays, and the second time period can include weekends. Or the first time period can include days, and the second time period can include evenings.
[0164] In accordance with a thirty-fifth aspect of the present inventions, a method of providing learning capability to a phone number calling system (such as, a phone), comprises receiving a plurality of phone numbers (e.g., those associated with incoming and/or outgoing phone calls), and maintaining a phone list containing the plurality of phone numbers and a plurality of priority values respectively associated with the plurality of phone numbers. The method further comprises selecting a set of phone numbers from the plurality of listed phone numbers based on the plurality of priority values, and communicating the phone number set to a user, e.g., by displaying it to the user. The selected phone number set can, e.g., be placed in an order according to corresponding priority values, e.g., the highest priority values. In the preferred method, a phone number probability containing the plurality of priority values is updated using a learning automaton or updated based purely on the frequency of the phone numbers, e.g., based on a total number of times the associated phone number is received during a specified time period. The method may further comprise selecting a phone number from the selected phone number set to make a phone call.
[0165] In accordance with a thirty-sixth of the present inventions, a method of providing learning capability to a television channel control system having an objective (e.g., anticipating watched television channels), comprises generating a list containing a plurality of listed television channels with an associated television channel probability distribution, which comprises a plurality of probability values corresponding to the plurality of listed television channels. Preferably, the television channel probability distribution is normalized.
[0166] The method further comprises selecting one or more television channels from the plurality of listed television channels based on the television channel probability distribution. In the preferred method, the selected television channel(s) corresponds to the highest probability values in the television channel probability distribution, and are placed in an order according to the corresponding probability values. In this manner, the “favorite” television channel(s) can be communicated to the user.
[0167] The method further comprises determining a performance index indicative of a performance of the television channel control system relative to its objective. For example, the method may comprise identifying a watched television channel, and determining if the identified television channel matches any listed television channels contained in the list and/or selected television channel(s). In this case, the performance index will be derived from this determination. The performance index may be instantaneous, e.g., if a currently identified television channel is used, or cumulative, e.g., if a tracked percentage of identified television channels is used.
[0168] The method further comprises modifying the television channel probability distribution based on the performance index. The television channel probability distribution can be modified in a variety of ways. For example, the television channel probability distribution can be modified by updating the television channel probability distribution, e.g., using a reward-inaction update. Or the television channel probability distribution can be modified by increasing a probability value corresponding to a particular listed television channel, or adding a probability value, e.g., when a new television channel is added to the list. In this case, the probability value can be replaced with the added probability value to, e.g., to minimize storage space.
[0169] In one preferred method, the television channel probability distribution is modified by updating it if the identified television channel matches any listed television channel. For example, the television channel probability distribution update can comprise a reward-inaction update, e.g., by rewarding the corresponding probability value. The method may further comprises adding a listed television channel corresponding to the identified television channel to the television channel list if the identified television channel does not match any listed television channel. In this case, the television channel probability distribution will be modified by adding a probability value corresponding to the added listed television channel to the television channel probability distribution. Another television channel on the television channel list can be replaced with the added listed television channel, and another probability value corresponding to the replaced listed television channel can be replaced with the added probability value.
[0170] In another preferred method, the television channel probability distribution is modified by updating it only if the identified television channel matches a television channel within the selected television channel(s). For example, the television channel probability distribution update can comprise a reward-inaction update, e.g., by rewarding the corresponding probability value. This preferred method may further comprise modifying the television channel probability distribution by increasing a corresponding probability value if the identified television channel matches a listed television channel that does not correspond to a television channel within the selected television channels. The method may further comprise adding a listed television channel corresponding to the identified television channel to the television channel list if the identified television channel does not match any listed television channel. In this case, the television channel probability distribution will be modified by adding a probability value corresponding to the added listed television channel to the television channel probability distribution. Another television channel on the television channel list can be replaced with the added listed television channel, and another probability value corresponding to the replaced listed television channel can be replaced with the added probability value.
[0171] The television channel probability distribution may optionally be updated using a learning automaton. A learning automaton can be characterized in that any given state of the action probability distribution determines the state of the next action probability distribution. That is, the next television channel probability distribution is a function of the current television channel probability distribution. Advantageously, updating of the television channel probability distribution using a learning automaton is based on a frequency of the television channels, as well as the time ordering of these television channels. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the television channel control system. Alternatively, the television channel probability distribution can be purely frequency-based. For example, television channel probability distribution can be based on a moving average.
[0172] The method may optionally comprise generating another television channel list containing at least another plurality of listed television channels and a television channel probability distribution comprising a plurality of probability values corresponding to the other plurality of listed television channels. This optional method further comprises selecting another set of television channels from the other plurality of television channels based on the other television channel probability distribution. A television channel associated with an action can then be identified, in which case, the method further comprises determining if the identified television channel matches any listed television channel contained in the television channel list. Another television channel associated with another action can also be identified, in which case, the method further comprises determining if the other identified television channel matches any listed television channel contained in the other television channel list. The performance index is this case will be derived from these matching determinations.
[0173] The two television channel lists can be used to distinguish between days of the week or time of day. For example, the method may further comprise identifying a television channel associated with an action, determining the current day of the week, selecting one of the two television channel lists based on the current day determination, and determining if the identified television channel matches any listed television channel contained in the selected television channel list. Or the method may further comprise identifying a television channel associated with another action, determining the current time of the day, selecting one of the two television channel lists based on the current time determination, and determining if the identified television channel matches any listed television channel contained in the selected television channel list.
[0174] Optionally, the television channel list can be one of a plurality of like television channel lists corresponding to a plurality of users, in which case, the method can further comprise determining which user watched the identified television channel, wherein the list corresponds with the determined user. Determination of the user can, e.g., be based on the operation of one of a plurality of keys associated with the television channel control system.
[0175] In accordance with a thirty-seventh aspect of the present inventions, a television channel control system having an objective (e.g., anticipating watched television channels) comprises a probabilistic learning module configured for learning favorite television channels of a user in response to identified watched television channels, and an intuition module configured for modifying a functionality of the probabilistic learning module based on the objective. The probabilistic learning module can include a learning automaton or can be purely frequency-based. The learning module and intuition module can be self-contained in a single device or distributed within several devices. For example, the learning module and intuition module can be contained with a remote control, a cable box, video cassette recorder, television, or any combination thereof.
[0176] Optionally, the probabilistic learning module is configured for determining a performance index indicative of a performance of the probabilistic learning module relative to the objective, and the intuition module is configured for modifying the probabilistic learning module functionality based on the performance index. As previously described, the performance index can be instantaneous or cumulative.
[0177] In the preferred embodiment, the probabilistic learning module comprises a television channel selection module configured for selecting the favorite television channel(s) from a plurality of television channels based on a television channel probability distribution comprising a plurality of probability values corresponding to the plurality of listed television channels. The favorite television channel(s) can correspond to the highest probability values in the television channel probability distribution. The television channel selection module can be further configured for placing the favorite numbers in an order according to corresponding probability values. The probabilistic learning module further comprises an outcome evaluation module configured for determining if identified television channels match any listed television channel contained in the television channel list, and a probability update module, wherein the intuition module is configured for modifying the probability update module based on the matching determinations.
[0178] The intuition module can modify the probability update module in a variety of ways. For example, the intuition module can be configured for modifying the probability update module by directing it to update the television channel probability distribution if any of the identified television channels matches any listed television channel. For example, a reward-inaction update can be used, e.g., by rewarding the corresponding probability value. The intuition module can further be configured for modifying the probability update module by adding a listed television channel corresponding to the identified television channel to the television channel list and adding a probability value corresponding to the added listed television channel to the television channel probability distribution if the identified television channel does not match any listed television channel. In this case, another television channel on the television channel list may be replaced with the added listed television channel, and another probability value corresponding to the replaced listed television channel can be replaced with the added probability value.
[0179] As another example, the intuition module can be configured for modifying the probability update module by directing it to update the television channel probability distribution only if the identified plurality of television channels matches a listed television channel corresponding to one of the favorite television channels. For example, a reward-inaction update can be used, e.g., by rewarding the corresponding probability value. The intuition module can further be configured for modifying the probability update module by increasing a corresponding probability value if the identified television channel matches a listed television channel that does not correspond to one of the favorite television channels. The intuition module can further be configured for modifying the probability update module by adding a listed television channel corresponding to the identified television channel to the television channel list and adding a probability value corresponding to the added listed television channel to the television channel probability distribution if the identified television channel does not match any listed television channel. In this case, another television channel on the television channel list may be replaced with the added listed television channel, and another probability value corresponding to the replaced listed television channel can be replaced with the added probability value.
[0180] In an optional embodiment, the favorite television channels can be divided into first and second favorite television channel lists, in which case, the probabilistic learning module can be configured for learning the first favorite television channel list in response to the identification of television channel associated television channels during a first time period, and for learning the second favorite television channel list in response to television channel associated television channels during a second time period. For example, the first time period can include weekdays, and the second time period can include weekends. Or the first time period can include days, and the second time period can include evenings.
[0181] In accordance with a thirty-eighth aspect of the present inventions, a method of providing learning capability to a television channel control system (e.g., a television remote control) having an objective (e.g., anticipating watched television channels), comprises generating a plurality of lists respectively corresponding to a plurality of television channel parameter values (e.g., switched channel numbers, channel types, channel age/gender, or channel rating). Each of the plurality of lists contains a plurality of listed television channels with an associated television channel probability distribution comprising a plurality of probability values corresponding to the plurality of listed television channels.
[0182] The method further comprises selecting a list corresponding to a parameter value exhibited by a currently identified television channel, and in the selected list, selecting one or more listed television channels from the plurality of listed television channels based on the television channel probability distribution. The method further comprises determining a performance index indicative of a performance of the television channel control system relative to its objective. For example, the method may comprise identifying an action associated television channel exhibiting a parameter value, selecting a list corresponding to the identified parameter value, and determining if the identified television channel matches any listed television channels contained in the selected list. In this case, the performance index will be derived from this determination. The performance index may be instantaneous, e.g., if a currently identified television channel is used, or cumulative, e.g., if a tracked percentage of identified television channels is used. The television channel probability distribution can be modified in a variety of ways, including those described above.
[0183] By way of non-limiting example, the use of a plurality of lists with respective associated parameter values allows an objective of the television channel control system (e.g., anticipating the favorite television channels of the user) to be better achieved by focussing on the list that more closely matches the television channel selection pattern that is currently exhibiting the corresponding parameter value.
[0184] In accordance with a thirty-ninth aspect of the present inventions, a method of providing learning capability to a processing device (e.g., an educational toy) comprises selecting one of a plurality of processor actions that is associated with a plurality of different difficulty levels. In the case of an educational toy, a selected action can be an educational game or an educational task to be performed by the user. Selection of the processor actions is based on an action probability distribution that contains a plurality of probability values corresponding to the plurality of processor actions. For example, the selected processor action can correspond to a pseudo-random selection of a value within the action probability distribution. Preferably, the action probability distribution is normalized. The method further comprises identifying an action performed by a user. In the preferred method, the user action is performed in response to the selected processor action.
[0185] The method further comprises determining an outcome of the selected processor action relative to the identified user action. For example, if the processing device is an educational toy, the outcome can be determined by determining if the identified user action matches a selected toy action. By way of non-limiting example, the outcome can be represented by one of two values (e.g., zero if the user is successful, and one if the user is not successful), one of a finite range of real numbers (e.g., lower numbers may mean user is relatively successful), or one of a range of continuous values (e.g., the lower the number, the more successful the user is).
[0186] Lastly, the method comprises updating the action probability distribution based on the outcome and the difficulty level of the selected processor action. By way of non-limiting example, if the outcome indicates that the identified user action is successful relative to the selected processor action, the action probability distribution can be shifted from one or more probability values corresponding to one or more processor actions associated with lesser difficulty levels to one or more probability values corresponding to one or more processor actions associated with greater difficulty levels. If the outcome indicates that the identified user action is unsuccessful relative to the selected processor action, the action probability distribution can be shifted from one or more probability values corresponding to one or more processor actions associated with greater difficulty levels to one or more probability values corresponding to one or more processor actions associated with lesser difficulty levels. In these cases, the one or more processor actions associated with the lesser difficulty levels preferably includes a processor action that is associated with a difficulty level equal to or greater than the difficulty level of the selected processor action, and the one or more processor actions associated with greater difficulty levels includes a processor action associated with a difficulty level equal to or greater than the difficulty level of the selected processor action.
[0187] A learning automaton can optionally be utilized to update the action probability distribution. A learning automaton can be characterized in that any given state of the game move probability distribution determines the state of the next action probability distribution. That is, the next action probability distribution is a function of the current action probability distribution. Advantageously, updating of the action probability distribution using a learning automaton is based on a frequency of the processor action and/or user action, as well as the time ordering of these processor actions. This can be contrasted with purely operating on a frequency of processor action or user actions, and updating the action probability distribution based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of educating the user. The action probability distribution can be updated using any of a variety of learning methodologies, e.g., linear or non-linear updates, absolutely expedient update, reward-penalty update, reward-inaction update, or inaction-penalty update.
[0188] In accordance with a fortieth aspect of the present inventions, a method of providing learning capability to a processing device having one or more objectives is provided. For example, if the processing device is an educational toy, the objective can be to increase the educational level of a user. The method comprises selecting one of a plurality of processor actions that is associated with a plurality of different difficulty levels, identifying an action performed by the user, and determining an outcome of the selected processor action relative to the identified user action. These steps can be performed in the manner previously described.
[0189] The method further comprises updating the action probability distribution based on the outcome and the difficulty level of the selected processor action, and modifying one or more of the processor action selection, outcome determination, and action probability distribution update based on the objective. The method may optionally comprise determining a performance index indicative of a performance of the education toy relative to the objective, in which case, the modification may be based on the performance index. The performance index may be derived from the outcome value and the difficulty level of the selected processor action. It may be cumulative or instantaneous.
[0190] In the preferred method, the modification comprises modifying the action probability distribution update, e.g., by selecting one of a predetermined plurality of learning methodologies employed by the action probability distribution update. By way of non-limiting example, if the outcome indicates that the identified user action is successful relative to the selected processor action, a learning methodology that rewards a processor action having a difficulty level equal to or greater than the difficulty level of the selected processor action can be selected, or a learning methodology that penalizes a processor action having a difficulty level equal to or less than said difficulty level of the selected processor action can be selected. If the outcome indicates that the identified user action is unsuccessful relative to the selected processor action, a learning methodology that rewards a processor action having a difficulty level equal to or less than the difficulty level of the selected processor action can be selected, or a learning methodology that penalizes a processor action having a difficulty level equal to or greater than the difficulty level of the selected processor action can be selected.
[0191] In accordance with a forty-first aspect of the present inventions, an educational toy having comprises a probabilistic learning module configured for learning a plurality of processor actions in response to a plurality of actions performed by a user. The educational toy further comprises an intuition module configured for modifying a functionality of the probabilistic learning module based on an objective of increasing the educational level of the user, e.g., by selecting one of a plurality of algorithms used by the learning module, or modifying a parameter of an algorithm employed by the learning module. The probabilistic learning module can include a learning automaton or can be purely frequency-based. The intuition module can optionally be further configured for determining a performance index indicative of a performance of the probabilistic learning module relative to the objective, and for modifying the probabilistic learning module functionality based on the performance index.
[0192] In the preferred embodiment, the probabilistic learning module may include an action selection module configured for selecting one of a plurality of processor actions associated with a plurality of different difficulty levels. The processor action selection can be based on an action probability distribution comprising a plurality of probability values corresponding to the plurality of processor action. The probabilistic learning module may further comprise an outcome evaluation module configured for determining an outcome of the selected processor action relative to the user action. The probabilistic learning module may further comprise a probability update module configured for updating the action probability distribution based on the outcome and the difficulty level of the selected processor action. When modifying the functionality of the learning module, the intuition module may modify a functionality of any combination of the game move selection module, outcome evaluation module, and probability update module.
[0193] In the preferred embodiment, the intuition module modifies the probability update module, e.g., by selecting one of a predetermined plurality of learning methodologies employed by the probability update module. For example, the intuition module can be configured for selecting a learning methodology that, if the outcome indicates that the identified user action is successful relative to the selected processor action, rewards a processor action having a difficulty level equal to or greater than the difficulty level of the selected processor action or penalizes a processor action having a difficulty level equal to or less than the difficulty level of the selected processor action. The intuition module can be further configured for selecting a learning methodology that, if the outcome indicates that the identified user action is unsuccessful relative to the selected processor action, rewards a processor action having a difficulty level equal to or less than the difficulty level of the selected processor action or penalizes a processor action having a difficulty level equal to or greater than the difficulty level of the selected processor action.
BRIEF DESCRIPTION OF THE DRAWINGS[0194] In order to better appreciate how the above-recited and other advantages and objects of the present inventions are obtained, a more particular description of the present inventions briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
[0195] FIG. 1 is a block diagram of a generalized single-user learning software program constructed in accordance with the present inventions, wherein a single-input, single output (SISO) model is assumed;
[0196] FIG. 2 is a diagram illustrating the generation of probability values for three actions over time in a prior art learning automaton;
[0197] FIG. 3 is a diagram illustrating the generation of probability values for three actions over time in the single-user learning software program of FIG. 1;
[0198] FIG. 4 is a flow diagram illustrating a preferred method performed by the program of FIG. 1;
[0199] FIG. 5 is a block diagram of a single-player duck hunting game to which the generalized program of FIG. 1 can be applied;
[0200] FIG. 6 is a plan view of a computer screen used in the duck hunting game of FIG. 5, wherein a gun is particularly shown shooting a duck;
[0201] FIG. 7 is a plan view of a computer screen used in the duck hunting game of FIG. 5, wherein a duck is particularly shown moving away from the gun;
[0202] FIG. 8 is a block diagram of a single-player game program employed in the duck hunting game of FIG. 5;
[0203] FIG. 9 is a flow diagram illustrating a preferred method performed by the game program of FIG. 8;
[0204] FIG. 10 is a flow diagram illustrating an alternative preferred method performed by the game program of FIG. 8;
[0205] FIG. 11 is a cartoon of a single-user educational child's toy to which the generalized program of FIG. 1 can be applied;
[0206] FIG. 12 is a block diagram of a single-user educational program employed in the educational child's toy of FIG. 11;
[0207] FIGS. 13a-13e are diagrams illustrating probability distribution modifications performed by the educational program of FIG. 12;
[0208] FIG. 14 is a flow diagram illustrating a preferred method performed by the educational program of FIG. 12;
[0209] FIG. 15 is a block diagram of another single-user educational program that can be employed in a modification of the educational child's toy of FIG. 11;
[0210] FIG. 16 is a flow diagram illustrating a preferred method performed by the educational program of FIG. 15;
[0211] FIG. 17 is a plan view of a mobile phone to which the generalized program of FIG. 1 can be applied;
[0212] FIG. 18 is a block diagram illustrating the components of the mobile phone of FIG. 17;
[0213] FIG. 19 is a block diagram of a priority listing program employed in the mobile phone of FIG. 17;
[0214] FIG. 20 is a flow diagram illustrating a preferred method performed by the priority listing program of FIG. 19;
[0215] FIG. 21 is a flow diagram illustrating an alternative preferred method performed by the priority listing program of FIG. 19;
[0216] FIG. 22 is a flow diagram illustrating still another preferred method performed by the priority listing program of FIG. 19;
[0217] FIG. 23 is a plan view of a television remote control unit to which the generalized program of FIG. 1 can be applied;
[0218] FIG. 24 is a block diagram illustrating the components of the remote control of FIG. 23;
[0219] FIG. 25 is a block diagram of a priority listing program employed in the remote control of FIG. 23;
[0220] FIG. 26 is a flow diagram illustrating a preferred method performed by the priority listing program of FIG. 25;
[0221] FIG. 27 is a plan view of another television remote control to which the generalized program of FIG. 1 can be applied;
[0222] FIG. 28 is a block diagram of a priority listing program employed in the remote control of FIG. 27;
[0223] FIG. 29 is a flow diagram illustrating a preferred method performed by the priority listing program of FIG. 28;
[0224] FIG. 30 is a block diagram of a generalized multiple-user learning software program constructed in accordance with the present inventions, wherein a single-input, multiple-output (SIMO) learning model is assumed;
[0225] FIG. 31 is a flow diagram a preferred method performed by the program of FIG. 30;
[0226] FIG. 32 is a block diagram of a multiple-player duck hunting game to which the generalized program of FIG. 30 can be applied, wherein the players simultaneously receive a single game move;
[0227] FIG. 33 is a block diagram of a multiple-player game program employed in the duck hunting game of FIG. 32;
[0228] FIG. 34 is a flow diagram illustrating a preferred method performed by the game program of FIG. 33;
[0229] FIG. 35 is a block diagram of another generalized multiple-user learning software program constructed in accordance with the present inventions, wherein a multiple-input, multiple-output (MIMO) learning model is assumed;
[0230] FIG. 36 is a flow diagram illustrating a preferred method performed by the program of FIG. 35;
[0231] FIG. 37 is a block diagram of a multiple-player duck hunting game to which the generalized program of FIG. 35 can be applied, wherein the players simultaneously receive multiple game moves;
[0232] FIG. 38 is a block diagram of a multiple-player game program employed in the duck hunting game of FIG. 37;
[0233] FIG. 39 is a flow diagram illustrating a preferred method performed by the game program of FIG. 38;
[0234] FIG. 40 is a block diagram of a first preferred computer system for distributing the processing power of the duck hunting game of FIG. 37;
[0235] FIG. 41 is a block diagram of a second preferred computer system for distributing the processing power of the duck hunting game of FIG. 37;
[0236] FIG. 42 is a block diagram of a third preferred computer system for distributing the processing power of the duck hunting game of FIG. 37;
[0237] FIG. 43 is a block diagram of a fourth preferred computer system for distributing the processing power of the duck hunting game of FIG. 37;
[0238] FIG. 44 is a block diagram of a fifth preferred computer system for distributing the processing power of the duck hunting game of FIG. 37;
[0239] FIG. 45 is a block diagram of still another generalized multiple-user learning software program constructed in accordance with the present inventions, wherein multiple SISO learning models are assumed;
[0240] FIG. 46 is a flow diagram illustrating a preferred method performed by the program of FIG. 45;
[0241] FIG. 47 is a block diagram of a multiple-player duck hunting game to which the generalized program of FIG. 45 can be applied;
[0242] FIG. 48 is a block diagram of a multiple-player game program employed in the duck hunting game of FIG. 47;
[0243] FIG. 49 is a flow diagram illustrating a preferred method performed by the game program of FIG. 48;
[0244] FIG. 50 is a block diagram illustrating the components of a mobile phone system to which the generalized program of FIG. 45 can be applied;
[0245] FIG. 51 is a block diagram of a priority listing program employed in the mobile phone system of FIG. 50;
[0246] FIG. 52 is a plan view of a television remote control to which the generalized program of FIG. 45 can be applied;
[0247] FIG. 53 is a block diagram of a priority listing program employed in the remote control of FIG. 52;
[0248] FIG. 54 is a flow diagram illustrating a preferred method performed by the priority listing program of FIG. 53;
[0249] FIG. 55 is a block diagram of yet another multiple-user learning software program constructed in accordance with the present inventions, wherein a maximum probability of majority approval (MPMA) learning model is assumed;
[0250] FIG. 56 is a flow diagram illustrating a preferred method performed by the program of FIG. 55;
[0251] FIG. 57 is a block diagram of a multiple-player game program that can be employed in the duck hunting game of FIG. 32 to which the generalized program of FIG. 55 can be applied;
[0252] FIG. 58 is a flow diagram illustrating a preferred method performed by the game program of FIG. 57;
[0253] FIG. 59 is a block diagram of a single-player game program that can be employed in a war game to which the generalized program of FIG. 55 can be applied;
[0254] FIG. 60 is a flow diagram illustrating a preferred method performed by the game program of FIG. 59;
[0255] FIG. 61 is a block diagram of a multiple-player game program that can be employed to generate revenue to which the generalized program of FIG. 55 can be applied;
[0256] FIG. 62 is a flow diagram illustrating a preferred method performed by the game program of FIG. 61;
[0257] FIG. 63 is a block diagram of yet another multiple-user learning software program constructed in accordance with the present inventions, wherein a maximum number of teachers approving (MNTA) learning model is assumed;
[0258] FIG. 64 is a flow diagram illustrating a preferred method performed by the program of FIG. 63;
[0259] FIG. 65 is a block diagram of a multiple-player game program that can be employed in the duck hunting game of FIG. 32 to which the generalized program of FIG. 63 can be applied;
[0260] FIG. 66 is a flow diagram illustrating a preferred method performed by the game program of FIG. 65;
[0261] FIG. 67 is a block diagram of yet another multiple-user learning software program constructed in accordance with the present inventions, wherein a teacher-action pair (TAP) learning model is assumed;
[0262] FIG. 68 is a flow diagram illustrating a preferred method performed by the program of FIG. 67;
[0263] FIG. 69 is a block diagram of a multiple-player game program that can be employed in the duck hunting game of FIG. 32 to which the generalized program of FIG. 67 can be applied; and
[0264] FIG. 70 is a flow diagram illustrating a preferred method performed by the game program of FIG. 69.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS[0265] Generalized Single-User Program (Single Processor Action-Single User Action)
[0266] Referring to FIG. 1, a single-user learning program 100 developed in accordance with the present inventions can be generally implemented to provide intuitive learning capability to any variety of processing devices, e.g., computers, microprocessors, microcontrollers, embedded systems, network processors, and data processing systems. In this embodiment, a single user 105 interacts with the program 100 by receiving a processor action &agr;x from a processor action set a within the program 100, selecting a user action &lgr;x from a user action set A based on the received processor action &agr;x, and transmitting the selected user action &lgr;x to the program 100. It should be noted that in alternative embodiments, the user 105 need not receive the processor action &agr;x to select a user action &lgr;x, the selected user action &lgr;x need not be based on the received processor action &agr;x, and/or the processor action &agr;x may be selected in response to the selected user action &lgr;x. The significance is that a processor action &agr;x and a user action &lgr;x are selected.
[0267] The program 100 is capable of learning based on the measured performance of the selected processor action &agr;x relative to a selected user action &lgr;x, which, for the purposes of this specification, can be measured as an outcome value &bgr;. It should be noted that although an outcome value &bgr; is described as being mathematically determined or generated for purposes of understanding the operation of the equations set forth herein, an outcome value &bgr; need not actually be determined or generated for practical purposes. Rather, it is only important that the outcome of the processor action &agr;x relative to the user action &lgr;xbe known. In alternative embodiments, the program 100 is capable of learning based on the measured performance of a selected processor action &agr;x and/or selected user action &lgr;x relative to other criteria. As will be described in further detail below, program 100 directs its learning capability by dynamically modifying the model that it uses to learn based on a performance index &phgr; to achieve one or more objectives.
[0268] To this end, the program 100 generally includes a probabilistic learning module 110 and an intuition module 115. The probabilistic learning module 110 includes a probability update module 120, an action selection module 125, and an outcome evaluation module 130. Briefly, the probability update module 120 uses learning automata theory as its learning mechanism with the probabilistic learning module 110 configured to generate and update an action probability distributions based on the outcome value &bgr;. The action selection module 125 is configured to pseudo-randomly select the processor action &agr;x based on the probability values contained within the action probability distribution p internally generated and updated in the probability update module 120. The outcome evaluation module 130 is configured to determine and generate the outcome value &bgr; based on the relationship between the selected processor action &agr;x and user action &lgr;x. The intuition module 115 modifies the probabilistic learning module 110 (e.g., selecting or modifying parameters of algorithms used in learning module 110) based on one or more generated performance indexes &phgr; to achieve one or more objectives. A performance index &phgr; can be generated directly from the outcome value &bgr; or from something dependent on the outcome value &bgr;, e.g., the action probability distribution p, in which case the performance index &phgr; may be a function of the action probability distribution p, or the action probability distribution p may be used as the performance index &phgr;. A performance index &phgr; can be cumulative (e.g., it can be tracked and updated over a series of outcome values &bgr; or instantaneous (e.g., a new performance index &phgr; can be generated for each outcome value &bgr;).
[0269] Modification of the probabilistic learning module 110 can be accomplished by modifying the functionalities of (1) the probability update module 120 (e.g., by selecting from a plurality of algorithms used by the probability update module 120, modifying one or more parameters within an algorithm used by the probability update module 120, transforming, adding and subtracting probability values to and from, or otherwise modifying the action probability distribution p); (2) the action selection module 125 (e.g., limiting or expanding selection of the action &agr; corresponding to a subset of probability values contained within the action probability distribution p); and/or (3) the outcome evaluation module 130 (e.g., modifying the nature of the outcome value &bgr; or otherwise the algorithms used to determine the outcome value &bgr;).
[0270] Having now briefly discussed the components of the program 100, we will now describe the functionality of the program 100 in more detail. Beginning with the probability update module 120, the action probability distribution p that it generates can be represented by the following equation:
p(k)=[px(k),p2(k),p3(k). . . pn(k)], [1]
[0271] where px is the action probability value assigned to a specific processor action &agr;x; n is the number of processor actions &agr;x within the processor action set &agr;x and k is the incremental time at which the action probability distribution was updated.
[0272] Preferably, the action probability distribution p at every time k should satisfy the following requirement: 1 ∑ i = 1 n p i ( k ) = 1 , 0 ≤ p i ( k ) ≤ 1. [ 2 ]
[0273] Thus, the internal sum of the action probability distribution p, i.e., the action probability values px for all processor actions &agr;x within the processor action set &agr; is always equal “1,” as dictated by the definition of probability. It should be noted that the number n of processor actions &agr;x need not be fixed, but can be dynamically increased or decreased during operation of the program 100.
[0274] The probability update module 120 uses a stochastic learning automaton, which is an automaton that operates in a random environment and updates its action probabilities in accordance with inputs received from the environment so as to improve its performance in some specified sense. A learning automaton can be characterized in that any given state of the action probability distribution p determines the state of the next action probability distribution p. For example, the probability update module 120 operates on the action probability distribution p(k) to determine the next action probability distributionp(k+1), i.e., the next action probability distribution p(k+1) is a function of the current action probability distribution p(k). Advantageously, updating of the action probability distribution p using a learning automaton is based on a frequency of the processor actions &agr;x and/or user actions &lgr;x, as well as the time ordering of these actions. This can be contrasted with purely operating on a frequency of processor actions &agr;x or user actions &lgr;x, and updating the action probability distribution p(k) based thereon. Although the present inventions, in their broadest aspects, should not be so limited, it has been found that the use of a learning automaton provides for a more dynamic, accurate, and flexible means of teaching the probabilistic learning module 110.
[0275] In this scenario, the probability update module 120 uses a single learning automaton with a single input to a single-teacher environment (with the user 105 as the teacher), and thus, a single-input, single-output (SISO) model is assumed.
[0276] To this end, the probability update module 120 is configured to update the action probability distribution p based on the law of reinforcement, the basic idea of which is to reward a favorable action and/or to penalize an unfavorable action. A specific processor action &agr;x is rewarded by increasing the corresponding current probability value px(k) and decreasing all other current probability values pj(k), while a specific processor action &agr;x is penalized by decreasing the corresponding current probability value px(k) and increasing all other current probability values pj(k). Whether the selected processor action &agr;x is rewarded or punished will be based on the outcome value &bgr; generated by the outcome evaluation module 130. For the purposes of this specification, an action probability distribution p is updated by changing the probability values px within the action probability distribution p, and does not contemplate adding or subtracting probability values px.
[0277] To this end, the probability update module 120 uses a learning methodology to update the action probability distribution p, which can mathematically be defined as:
p(k+1)=T[p(k),&agr;x(k),&bgr;(k)], [3]
[0278] where p(k+1) is the updated action probability distribution, T is the reinforcement scheme, p(k) is the current action probability distribution, &agr;x(k) is the previous processor action, &bgr;(k) is latest outcome value, and k is the incremental time at which the action probability distribution was updated.
[0279] Alternatively, instead of using the immediately previous processor action a,(k), any set of previous processor action, e.g., &agr;(k−1), &agr;(k−2), &agr;(k−3), etc., can be used for lag learning, and/or a set of future processor action, e.g., &agr;(k+1), &agr;(k+2), &agr;(k+3), etc., can be used for lead learning. In the case of lead learning, a future processor action is selected and used to determine the updated action probability distribution p(k+1).
[0280] The types of learning methodologies that can be utilized by the probability update module 120 are numerous, and depend on the particular application. For example, the nature of the outcome value &bgr; can be divided into three types: (1) P-type, wherein the outcome value &bgr; can be equal to “1” indicating success of the processor action &agr;x, and “0” indicating failure of the processor action &agr;x; (2) Q-type, wherein the outcome value &bgr; can be one of a finite number of values between “0” and “1” indicating a relative success or failure of the processor action &agr;x; or (3) S-Type, wherein the outcome value &bgr; can be a continuous value in the interval [0,1] also indicating a relative success or failure of the processor action &agr;x.
[0281] The outcome value &bgr; can indicate other types of events besides successful and unsuccessful events. The time dependence of the reward and penalty probabilities of the actions a can also vary. For example, they can be stationary if the probability of success for a processor action &agr;x does not depend on the index k, and non-stationary if the probability of success for the processor action &agr;x depends on the index k. Additionally, the equations used to update the action probability distribution p can be linear or non-linear. Also, a processor action &agr;x can be rewarded only, penalized only, or a combination thereof. The convergence of the learning methodology can be of any type, including ergodic, absolutely expedient, or optimal, or optimal. The learning methodology can also be a discretized, estimator, pursuit, hierarchical, pruning, growing or any combination thereof.
[0282] Of special importance is the estimator learning methodology, which can advantageously make use of estimator tables and algorithms should it be desired to reduce the processing otherwise requiring for updating the action probability distribution for every processor action &agr;x that is received. For example, an estimator table may keep track of the number of successes and failures for each processor action &agr;x received, and then the action probability distribution p can then be periodically updated based on the estimator table by, e.g., performing transformations on the estimator table. Estimator tables are especially useful when multiple users are involved, as will be described with respect to the multi-user embodiments described later.
[0283] In the preferred embodiment, a reward function gj and a penalization function hj is used to accordingly update the current action probability distribution p(k). For example, a general updating scheme applicable to P-type, Q-type and S-type methodologies can be given by the following SISO equations:
pj(k+1)=pj(k)−&bgr;(k)gj(pj(k))+(1−&bgr;(k))hj(p(k)), if &agr;(k)≠&agr;x [4] 2 p i ( k + 1 ) = p i ( k ) + β ( k ) ∑ j = 1 j ≠ i n g j ( p ( k ) ) - ( 1 - β ( k ) ) ∑ j = 1 j ≠ i n h j ( p ( k ) ) , if α ( k ) = α i [ 5 ]
[0284] where i is an index for a processor action &agr;x, selected to be rewarded or penalized, and j is an index for the remaining processor actions &agr;x
[0285] Assuming a P-type methodology, equations [4] and [5] can be broken down into the following equations: 3 p i ( k + 1 ) = p i ( k ) + ∑ j = 1 j ≠ i n g j ( p ( k ) ) ; and [ 6 ]
pj(k+1)=pj(k)−gj(p(k)), when &bgr;(k)=1 and &agr;x is selected [7] 4 p i ( k + 1 ) = p i ( k ) - ∑ j = 1 j ≠ i n h j ( p ( k ) ) ; and [ 8 ]
pj(k+1)=pj(k)+hj(p(k)), when &bgr;(k)=0 and &agr;x is selected [9]
[0286] Preferably, the reward function gj and penalty function hj are continuous and nonnegative for purposes of mathematical convenience and to maintain the reward and penalty nature of the updating scheme. Also, the reward function gj and penalty function hj are preferably constrained by the following equations to ensure that all of the components of p(k+1) remain in the (0, 1) interval when p(k) is in the (0,1) interval:
0<gj(p)<pj; 5 0 < ∑ j ≠ i n ( p j + h j ( p ) ) < 1
[0287] for all pj &egr;(0,1) and all j=1, 2, . . . n.
[0288] The updating scheme can be of the reward-penalty type, in which case, both gj and hj are non-zero. Thus, in the case of a P-type methodology, the first two updating equations [6] and [7] will be used to reward the processor action &agr;x, e.g., when successful, and the last two updating equations [8] and [9] will be used to penalize processor action &agr;x, e.g., when unsuccessful. Alternatively, the updating scheme is of the reward-inaction type, in which case, gj is nonzero and hj is zero. Thus, the first two general updating equations [6] and [7] will be used to reward the processor action &agr;x, e.g., when successful, whereas the last two general updating equations [8] and [9] will not be used to penalize processor action &agr;x, e.g., when unsuccessful. More alternatively, the updating scheme is of the penalty-inaction type, in which case, gj is zero and hj is nonzero. Thus, the first two general updating equations [6] and [7] will not be used to reward the processor action &agr;x, e.g., when successful, whereas the last two general updating equations [8] and [9] will be used to penalize processor action &agr;x, e.g., when unsuccessful. The updating scheme can even be of the reward-reward type (in which case, the processor action &agr;x is rewarded more, e.g., when it is more successful than when it is not) or penalty-penalty type (in which case, the processor action &agr;x is penalized more, e.g., when it is less successful than when it is).
[0289] It should be noted that with respect to the probability distribution p as a whole, any typical updating scheme will have both a reward aspect and a penalty aspect to the extent that a particular processor action &agr;x that is rewarded will penalize the remaining processor actions &agr;x, and any particular processor action &agr;x that penalized will reward the remaining processor actions &agr;i. This is because any increase in a probability value px will relatively decrease the remaining probability values px, and any decrease in a probability value px will relatively increase the remaining probability values px. For the purposes of this specification, however, a particular processor action &agr;x is only rewarded if its corresponding probability value px is increased in response to an outcome value &bgr; associated with it, and a processor action &agr;x is only penalized if its corresponding probability value px is decreased in response to an outcome value &bgr; associated with it.
[0290] The nature of the updating scheme is also based on the functions gj and hj themselves. For example, the functions gj and hj can be linear, in which case, e.g., they can be characterized by the following equations:
gj(p(k))=apj(k), 0<&agr;<1; and [10] 6 h j ( p ( k ) ) = b n - 1 - bp j ( k ) , 0 < b < 1 [ 11 ]
[0291] where &agr; is the reward parameter, and b is the penalty parameter.
[0292] The functions gj and hj can alternatively be absolutely expedient, in which case, e.g., they can be characterized by the following equations: 7 g 1 ( p ) p 1 = g 2 ( p ) p 2 = … = g n ( p ) p n ; [ 12 ] h 1 ( p ) p 1 = h 2 ( p ) p 2 = … = h n ( p ) p n [ 13 ]
[0293] The functions gj and hj can alternatively be non-linear, in which case, e.g., they can be characterized by the following equations:
gj(p(k))=pj(k)−F(pj(k)); 8 h j ( p ( k ) ) = p i ( k ) - F ( p i ( k ) ) n - 1 [ 15 ]
[0294] and F(x)=axm,m=2, 3, . . . .
[0295] It should be noted that equations [4] and [5] are not the only general equations that can be used to update the current action probability distribution p(k) using a reward function gj and a penalization function hj. For example, another general updating scheme applicable to P-type, Q-type and S-type methodologies can be given by the following SISO equations:
pj(k+1)=pj(k)−&bgr;(k)cjgx(p(k))+(1−&bgr;(k))djxhx(p(k)), if &agr;(k)≠&agr;x [16]
p(k+1)=px(k)+&bgr;(k)gi(p(k))−(1−&bgr;(k))hx(p(k)), if &agr;(k)=&agr;i
[0296] where c and d are constant or variable distribution multipliers that adhere to the following constraints: 9 ∑ j = 1 i ≠ j n c j g i ( p ( k ) ) = g i ( p ( k ) ) ; ∑ j = 1 i ≠ j n d j h i ( p ( k ) ) = h i ( p ( k ) )
[0297] In other words, the multipliers c and d are used to determine what proportions of the amount that is added to or subtracted from the probability value px is redistributed to the remaining probability values pj.
[0298] Assuming a P-type methodology, equations [16] and [17] can be broken down into the following equations:
px(k+1)=px(k)+gx(p(k)); and [18]
pj(k+1)=pj(k)−cjgx(p(k)), when &bgr;(k)=1 and &agr;x is selected [19]
px(k+1)=px(k)−hx(p(k)); and [20]
pj(k+1)=pj(k)+djhx(p(k)), when &bgr;(k)=0 and &agr;x is selected [20]
[0299] It can be appreciated that equations [4]-[5] and [16]-[17] are fundamentally similar to the extent that the amount that is added to or subtracted from the probability value px is subtracted from or added to the remaining probability values pj. The fundamental difference is that, in equations [4]-[5], the amount that is added to or subtracted from the probability value px is based on the amounts that are subtracted from or added to the remaining probability values pj (i.e., the amounts added to or subtracted from the remaining probability values pij are calculated first), whereas in equations [16]-[17], the amounts that are added to or subtracted from the remaining probability values pj are based on the amount that is subtracted from or added to the probability value px (i.e., the amount added to or subtracted from the probability value px is calculated first). It should also be noted that equations [4]-[5] and [16]-[17] can be combined to create new learning methodologies. For example, the reward portions of equations [4]-[5] can be used when an action &agr;x is to be rewarded, and the penalty portions of equations [16]-[17] can be used when an action &agr;x is to be penalized.
[0300] Previously, the reward and penalty functions gj and hj and multipliers cj and dj have been described as being one-dimensional with respect to the current action &agr;x that is being rewarded or penalized. That is, the reward and penalty functions gj and hj and multipliers cj and dj are the same given any action &agr;x. It should be noted, however, that multi-dimensional reward and penalty functions gijx and hijx and multipliers cijx and dijx can be used.
[0301] In this case, the single dimensional reward and penalty functions gj and hj of equations [6]-[9] can be replaced with the two-dimensional reward and penalty functions gijx and hijx, resulting in the following equations: 10 p i ( k + 1 ) = p i ( k ) + ∑ j = 1 j ≠ i n g ij ( p ( k ) ) ; and [6a] p j ( k + 1 ) = p j ( k ) - g ij ( p ( k ) ) , when β ( k ) = 1 and α i is selected [7a] p i ( k + 1 ) = p i ( k ) - ∑ j = 1 j ≠ i n h ij ( p ( k ) ) ; and [8a]
pj(k+1)=pj(k)+hijx(p(k)), when &bgr;(k)=0 and &agr;x is selected [9a]
[0302] The single dimensional multipliers cj and dj of equations [1 9] and [21] can be replaced with the two-dimensional multipliers cijx and dijx, resulting in the following equations:
pj(k+1)=pj(k)−cijxgx(p(k)), when &bgr;(k)=1 and &agr;x is selected [19a]
pj(k+1)=pj(k)+dijxhx(p(k)), when &bgr;(k)=0 and &agr;x is selected [21a]
[0303] Thus, it can be appreciated, that equations [19a] and [21a] can be expanded into many different learning methodologies based on the particular action &agr;x that has been selected.
[0304] Further details on learning methodologies are disclosed in “Learning Automata An Introduction,” Chapter 4, Narendra, Kumpati, Prentice Hall (1989) and “Learning Algorithms-Theory and Applications in Signal Processing, Control and Communications,” Chapter 2, Mars, Phil, CRC Press (1996), which are both expressly incorporated herein by reference.
[0305] The intuition module 115 directs the learning of the program 100 towards one or more objectives by dynamically modifying the probabilistic learning module 110. The intuition module 115 specifically accomplishes this by operating on one or more of the probability update module 120, action selection module 125, or outcome evaluation module 130 based on the performance index &phgr;, which, as briefly stated, is a measure of how well the program 100 is performing in relation to the one or more objective to be achieved. The intuition module 115 may, e.g., take the form of any combination of a variety of devices, including an (1) evaluator, data miner, analyzer, feedback device, stabilizer; (2) decision maker; (3) expert or rule-based system; (4) artificial intelligence, fuzzy logic, neural network, or genetic methodology; (5) directed learning device; (6) statistical device, estimator, predictor, regressor, or optimizer. These devices may be deterministic, pseudo-deterministic, or probabilistic.
[0306] It is worth noting that absent modification by the intuition module 115, the probabilistic learning module 110 would attempt to determine a single best action or a group of best actions for a given predetermined environment as per the objectives of basic learning automata theory. That is, if there is a unique action that is optimal, the unmodified probabilistic learning module 110 will substantially converge to it. If there is a set of actions that are optimal, the unmodified probabilistic learning module 110 will substantially converge to one of them, or oscillate (by pure happenstance) between them. In the case of a changing environment, however, the performance of an unmodified learning module 110 would ultimately diverge from the objectives to be achieved. FIGS. 2 and 3 are illustrative of this point. Referring specifically to FIG. 2, a graph illustrating the action probability values px of three different actions &agr;1, &agr;2, and &agr;3, as generated by a prior art learning automaton over time t, is shown. As can be seen, the action probability values px for the three actions are equal at the beginning of the process, and meander about on the probability plane px until they eventually converge to unity for a single action, in this case, &agr;1. Thus, the prior art learning automaton assumes that there is always a single best action over time t and works to converge the selection to this best action. Referring specifically to FIG. 3, a graph illustrating the action probability values px of three different actions &agr;1, &agr;2, and &agr;3, as generated by the program 100 over time t, is shown. Like with the prior art learning automaton, action probability values px for the three action are equal at t=0. Unlike with the prior art learning automaton, however, the action probability values px for the three actions meander about on the probability plane p without ever converging to a single action. Thus, the program 100 does not assume that there is a single best action over time t, but rather assumes that there is a dynamic best action that changes over time t. Because the action probability value for any best action will not be unity, selection of the best action at any given time t is not ensured, but will merely tend to occur, as dictated by its corresponding probability value. Thus, the program 100 ensures that the objective(s) to be met are achieved over time t.
[0307] Having now described the interrelationships between the components of the program 100 and the user 105, we now generally describe the methodology of the program 100. Referring to FIG. 4, the action probability distributions is initialized (step 150). Specifically, the probability update module 120 initially assigns equal probability values to all processor actions &agr;x in which case, the initial action probability distribution p(k) can be represented by px(0)=p2(0)=p2(0)= . . . pn(0)=
[0308] Thus, each of the processor actions &agr;x has an equal chance of being selected by the action selection module 125. Alternatively, the probability update module 120 initially assigns unequal probability values to at least some of the processor actions &agr;i, e.g., if the programmer desires to direct the learning of the program 100 towards one or more objectives quicker. For example, if the program 100 is a computer game and the objective is to match a novice game player's skill level, the easier processor action &agr;x, and in this case game moves, may be assigned higher probability values, which as will be discussed below, will then have a higher probability of being selected. In contrast, if the objective is to match an expert game player's skill level, the more difficult game moves may be assigned higher probability values.
[0309] Once the action probability distributionp is initialized at step 150, the action selection module 125 determines if a user action &lgr;x has been selected from the user action set &lgr; (step 155). If not, the program 100 does not select a processor action &agr;x from the processor action set a (step 160), or alternatively selects a processor action &agr;x, e.g., randomly, notwithstanding that a user action &lgr;x has not been selected (step 165), and then returns to step 155 where it again determines if a user action &lgr;x has been selected. If a user action &lgr;x as been selected at step 155, the action selection module 125 determines the nature of the selected user action &lgr;x, i.e., whether the selected user action &lgr;x is of the type that should be countered with a processor action &agr;x and/or whether the performance index &phgr; can be based, and thus whether the action probability distribution p should be updated. For example, again, if the program 100 is a game program, e.g., a shooting game, a selected user action &lgr;x that merely represents a move may not be a sufficient measure of the performance index &phgr;, but should be countered with a processor action &agr;i, while a selected user action Ax that represents a shot may be a sufficient measure of the performance index &phgr;.
[0310] Specifically, the action selection module 125 determines whether the selected user action &lgr;x is of the type that should be countered with a processor action &agr; (step 170). If so, the action selection module 125 selects a processor action &agr;x from the processor action set a based on the action probability distributions (step 175). After the performance of step 175 or if the action selection module 125 determines that the selected user action Ax is not of the type that should be countered with a processor action &agr;x, the action selection module 125 determines if the selected user action &lgr;x is of the type that the performance index &phgr; is based on (step 180).
[0311] If so, the outcome evaluation module 130 quantifies the performance of the previously selected processor action &agr;x (or a more previous selected processor action &agr;x in the case of lag learning or a future selected processor action &agr;x in the case of lead learning) relative to the currently selected user action &lgr;x by generating an outcome value &bgr; (step 185). The intuition module 115 then updates the performance index &phgr; based on the outcome value &agr;, unless the performance index &phgr; is an instantaneous performance index that is represented by the outcome value &bgr; itself (step 190). The intuition module 115 then modifies the probabilistic learning module 110 by modifying the functionalities of the probability update module 120, action selection module 125, or outcome evaluation module 130 (step 195). It should be noted that step 190 can be performed before the outcome value &bgr; is generated by the outcome evaluation module 130 at step 180, e.g., if the intuition module 115 modifies the probabilistic learning module 110 by modifying the functionality of the outcome evaluation module 130. The probability update module 120 then, using any of the updating techniques described herein, updates the action probability distribution p based on the generated outcome value &bgr; (step 198).
[0312] The program 100 then returns to step 155 to determine again whether a user action &lgr;x has been selected from the user action set &lgr;. It should be noted that the order of the steps described in FIG. 4 may vary depending on the specific application of the program 100.
[0313] Single-Player Game Program (Single Game Move-Single Player Move)
[0314] Having now generally described the components and functionality of the learning program 100, we now describe one of its various applications. Referring to FIG. 5, a single-player game program 300 (shown in FIG. 8) developed in accordance with the present inventions is described in the context of a duck hunting game 200. The game 200 comprises a computer system 205, which, e.g., takes the form of a personal desktop or laptop computer. The computer system 205 includes a computer screen 210 for displaying the visual elements of the game 200 to a player 215, and specifically, a computer animated duck 220 and a gun 225, which is represented by a mouse cursor. For the purposes of this specification, the duck 220 and gun 225 can be broadly considered to be computer and user-manipulated objects, respectively. The computer system 205 further comprises a computer console 250, which includes memory 230 for storing the game program 300, and a CPU 235 for executing the game program 300. The computer system 205 further includes a computer mouse 240 with a mouse button 245, which can be manipulated by the player 215 to control the operation of the gun 225, as will be described immediately below. It should be noted that although the game 200 has been illustrated as being embodied in a standard computer, it can very well be implemented in other types of hardware environments, such as a video game console that receives video game cartridges and connects to a television screen, or a video game machine of the type typically found in video arcades.
[0315] Referring specifically to the computer screen 210 of FIGS. 6 and 7, the rules and objective of the duck hunting game 200 will now be described. The objective of the player 215 is to shoot the duck 220 by moving the gun 225 towards the duck 220, intersecting the duck 220 with the gun 225, and then firing the gun 225 (FIG. 6). The player 215 accomplishes this by laterally moving the mouse 240, which correspondingly moves the gun 225 in the direction of the mouse movement, and clicking the mouse button 245, which fires the gun 225. The objective of the duck 220, on the other hand, is to avoid from being shot by the gun 225. To this end, the duck 220 is surrounded by a gun detection region 270, the breach of which by the gun 225 prompts the duck 220 to select and make one of seventeen moves 255 (eight outer moves 255a, eight inner moves 255b, and a non-move) after a preprogrammed delay (move 3 in FIG. 7). The length of the delay is selected, such that it is not so long or short as to make it too easy or too difficult to shoot the duck 220. In general, the outer moves 255a more easily evade the gun 225 than the inner moves 255b, thus, making it more difficult for the player 215 to shot the duck 220.
[0316] For purposes of this specification, the movement and/or shooting of the gun 225 can broadly be considered to be a player move, and the discrete moves of the duck 220 can broadly be considered to be computer or game moves, respectively. Optionally or alternatively, different delays for a single move can also be considered to be game moves. For example, a delay can have a low and high value, a set of discrete values, or a range of continuous values between two limits. The game 200 maintains respective scores 260 and 265 for the player 215 and duck 220. To this end, if the player 215 shoots the duck 220 by clicking the mouse button 245 while the gun 225 coincides with the duck 220, the player score 260 is increased. In contrast, if the player 215 fails to shoot the duck 220 by clicking the mouse button 245 while the gun 225 does not coincide with the duck 220, the duck score 265 is increased. The increase in the score can be fixed, one of a multitude of discrete values, or a value within a continuous range of values.
[0317] As will be described in further detail below, the game 200 increases its skill level by learning the player's 215 strategy and selecting the duck's 220 moves based thereon, such that it becomes more difficult to shoot the duck 220 as the player 215 becomes more skillful. The game 200 seeks to sustain the player's 215 interest by challenging the player 215. To this end, the game 200 continuously and dynamically matches its skill level with that of the player 215 by selecting the duck's 220 moves based on objective criteria, such as, e.g., the difference between the respective player and game scores 260 and 265. In other words, the game 200 uses this score difference as a performance index &phgr; in measuring its performance in relation to its objective of matching its skill level with that of the game player. In the regard, it can be said that the performance index &phgr; is cumulative. Alternatively, the performance index &phgr; can be a function of the game move probability distribution p.
[0318] Referring further to FIG. 8, the game program 300 generally includes a probabilistic learning module 310 and an intuition module 315, which are specifically tailored for the game 200. The probabilistic learning module 310 comprises a probability update module 320, a game move selection module 325, and an outcome evaluation module 330. Specifically, the probability update module 320 is mainly responsible for learning the player's 215 strategy and formulating a counterstrategy based thereon, with the outcome evaluation module 330 being responsible for evaluating moves performed by the game 200 relative to moves performed by the player 215. The game move selection module 325 is mainly responsible for using the updated counterstrategy to move the duck 220 in response to moves by the gun 225. The intuition module 315 is responsible for directing the learning of the game program 300 towards the objective, and specifically, dynamically and continuously matching the skill level of the game 200 with that of the player 215. In this case, the intuition module 315 operates on the game move selection module 325, and specifically selects the methodology that the game move selection module 325 will use to select a game move &agr;x from the game move set a as will be discussed in further detail below. In the preferred embodiment, the intuition module 315 can be considered deterministic in that it is purely rule-based. Alternatively, however, the intuition module 315 can take on a probabilistic nature, and can thus be quasi-deterministic or entirely probabilistic.
[0319] To this end, the game move selection module 325 is configured to receive a player move &lgr;1x from the player 215, which takes the form of a mouse 240 position, i.e., the position of the gun 225, at any given time. In this embodiment, the player move &lgr;1x can be selected from a virtually infinite player move set &lgr;1x, i.e., the number of player moves &lgr;1x are only limited by the resolution of the mouse 240. Based on this, the game move selection module 325 detects whether the gun 225 is within the detection region 270, and if so, selects a game move &agr;x from the game move set &agr;x and specifically, one of the seventeen moves 255 that the duck 220 can make. The game move &agr;x manifests itself to the player 215 as a visible duck movement.
[0320] The game move selection module 325 selects the game move &agr;x based on the updated game strategy. To this end, the game move selection module 325 is further configured to receive the game move probability distribution p from the probability update module 320, and pseudo-randomly selecting the game move &agr;x based thereon. The game move probability distribution p is similar to equation [1] and can be represented by the following equation:
p(k)=[p1(k),p2(k),p3(k) . . . pn(k)], [1-1]
[0321] where px is the game move probability value assigned to a specific game move &agr;i; n is the number of game moves &agr;x within the game move set &agr;x and k is the incremental time at which the game move probability distribution was updated.
[0322] It is noted that pseudo-random selection of the game move &agr;x allows selection and testing of any one of the game moves &agr;x, with those game moves &agr;x corresponding to the highest probability values being selected more often. Thus, without the modification, the game move selection module 325 will tend to more often select the game move &agr;x to which the highest probability value px corresponds, so that the game program 300 continuously improves its strategy, thereby continuously increasing its difficulty level.
[0323] Because the objective of the game 200 is sustainability, i.e., dynamically and continuously matching the respective skill levels of the game 200 and player 215, the intuition module 315 is configured to modify the functionality of the game move selection module 325 based on the performance index &phgr;, and in this case, the current skill level of the player 215 relative to the current skill level of the game 200. In the preferred embodiment, the performance index &phgr; is quantified in terms of the score difference value &Dgr; between the player score 260 and the duck score 265. The intuition module 315 is configured to modify the functionality of the game move selection module 325 by subdividing the game move set &agr; into a plurality of game move subsets &agr;s, one of which will be selected by the game move selection module 325. In an alternative embodiment, the game move selection module 325 may also select the entire game move set &agr;. In another alternative embodiment, the number and size of the game move subsets &agr;s can be dynamically determined.
[0324] In the preferred embodiment, if the score difference value &Dgr; is substantially positive (i.e., the player score 260 is substantially higher than the duck score 265), the intuition module 315 will cause the game move selection module 325 to select a game move subset &agr;x, the corresponding average probability value of which will be relatively high, e.g., higher than the median probability value of the game move probability distribution p. As a further example, a game move subset &agr;s corresponding to the highest probability values within the game move probability distribution p can be selected. In this manner, the skill level of the game 200 will tend to quickly increase in order to match the player's 215 higher skill level.
[0325] If the score difference value &Dgr; is substantially negative (i.e., the player score 260 is substantially lower than the duck score 265), the intuition module 315 will cause the game move selection module 325 to select a game move subset &agr;s, the corresponding average probability value of which will be relatively low, e.g., lower than the median probability value of the game move probability distribution p. As a further example, a game move subset &agr;s, corresponding to the lowest probability values within the game move probability distribution p can be selected. In this manner, the skill level of the game 200 will tend to quickly decrease in order to match the player's 215 lower skill level.
[0326] If the score difference value A is substantially low, whether positive or negative (i.e., the player score 260 is substantially equal to the duck score 265), the intuition module 315 will cause the game move selection module 325 to select a game move subset &agr;s, the average probability value of which will be relatively medial, e.g., equal to the median probability value of the game move probability distribution p. In this manner, the skill level of the game 200 will tend to remain the same, thereby continuing to match the player's 215 skill level. The extent to which the score difference value A is considered to be losing or winning the game 200 may be provided by player feedback and the game designer.
[0327] Alternatively, rather than selecting a game move subset &agr;s, based on a fixed reference probability value, such as the median probability value of the game move probability distribution p, selection of the game move set &agr;s can be based on a dynamic reference probability value that moves relative to the score difference value &Dgr;. To this end, the intuition module 315 increases and decreases the dynamic reference probability value as the score difference value A becomes more positive or negative, respectively. Thus, selecting a game move subset &agr;x, the corresponding average probability value of which substantially coincides with the dynamic reference probability value, will tend to match the skill level of the game 200 with that of the player 215. Without loss of generality, the dynamic reference probability value can also be learning using the learning principles disclosed herein.
[0328] In the illustrated embodiment, (1) if the score difference value &Dgr; is substantially positive, the intuition module 315 will cause the game move selection module 325 to select a game move subset &agr;s composed of the top five corresponding probability values; (2) if the score difference value &Dgr; is substantially negative, the intuition module 315 will cause the game move selection module 325 to select a game move subset &agr;s composed of the bottom five corresponding probability values; and (3) if the score difference value &Dgr; is substantially low, the intuition module 315 will cause the game move selection module 325 to select a game move subset &agr;s composed of the middle seven corresponding probability values, or optionally a game move subset &agr;s composed of all seventeen corresponding probability values, which will reflect a normal game where all game moves are available for selection.
[0329] Whether the reference probability value is fixed or dynamic, hysteresis is preferably incorporated into the game move subset &agr;s selection process by comparing the score difference value A to upper and lower score difference thresholds NS1 and NS2, e.g., −1000 and 1000, respectively. Thus, the intuition module 315 will cause the game move selection module 325 to select the game move subset &agr;s in accordance with the following criteria:
[0330] If &Dgr;<NS1, then select game move subset &agr;s with relatively low probability values;
[0331] If &Dgr;>NS2, then select game move subset &agr;s with relatively high probability values; and
[0332] If NS1≦&Dgr;≦NS2, then select game move subset &agr;s with relatively medial probability values.
[0333] Alternatively, rather than quantify the relative skill level of the player 215 in terms of the score difference value &Dgr; between the player score 260 and the duck score 265, as just previously discussed, the relative skill level of the player 215 can be quantified from a series (e.g., ten) of previous determined outcome values &bgr;. For example, if a high percentage of the previous determined outcome values &bgr; is equal to “0,” indicating a high percentage of unfavorable game moves &agr;i, the relative player skill level can be quantified as be relatively high. In contrast, if a low percentage of the previous determined outcome values &bgr; is equal to “0,” indicating a low percentage of unfavorable game moves &agr;x, the relative player skill level can be quantified as be relatively low. Thus, based on this information, a game move &agr;x can be pseudo-randomly selected, as hereinbefore described.
[0334] The game move selection module 325 is configured to pseudo-randomly select a single game move &agr;x from the game move subset &agr;s, thereby minimizing a player detectable pattern of game move &agr;x selections, and thus increasing interest in the game 200. Such pseudo-random selection can be accomplished by first normalizing the game move subset &agr;s, and then summing, for each game move &agr;i within the game move subset &agr;s, the corresponding probability value with the preceding probability values (for the purposes of this specification, this is considered to be a progressive sum of the probability values). For example, the following Table 1 sets forth the unnormalized probability values, normalized probability values, and progressive sum of an exemplary subset of five game moves: 1 TABLE 1 Progressive Sum of Probability Values For Five Exemplary Game Moves in SISO Format Unnormalized Normalized Progressive Game Move Probability Value Probability Value Sum &agr;1 0.05 0.09 0.09 &agr;2 0.05 0.09 0.18 &agr;3 0.10 0.18 0.36 &agr;4 0.15 0.27 0.63 &agr;5 0.20 0.37 1.00
[0335] The game move selection module 325 then selects a random number between “0” and “1,” and selects the game move &agr;x corresponding to the next highest progressive sum value. For example, if the randomly selected number is 0.38, game move &agr;4 will be selected.
[0336] The game move selection module 325 is further configured to receive a player move &lgr;2x from the player 215 in the form of a mouse button 245 click/mouse 240 position combination, which indicates the position of the gun 225 when it is fired. The outcome evaluation module 330 is configured to determine and output an outcome value &bgr; that indicates how favorable the game move &agr;x is in comparison with the received player move &lgr;2x.
[0337] To determine the extent of how favorable a game move &agr;x is, the outcome evaluation module 330 employs a collision detection technique to determine whether the duck's 220 last move was successful in avoiding the gunshot. Specifically, if the gun 225 coincides with the duck 220 when fired, a collision is detected. On the contrary, if the gun 225 does not coincide with the duck 220 when fired, a collision is not detected. The outcome of the collision is represented by a numerical value, and specifically, the previously described outcome value &bgr;. In the illustrated embodiment, the outcome value &bgr; equals one of two predetermined values: “1” if a collision is not detected (i.e., the duck 220 is not shot), and “0” if a collision is detected (i.e., the duck 220 is shot). Of course, the outcome value &bgr; can equal “0” if a collision is not detected, and “1” if a collision is detected, or for that matter one of any two predetermined values other than a “0” or “1,” without straying from the principles of the invention. In any event, the extent to which a shot misses the duck 220 (e.g., whether it was a near miss) is not relevant, but rather that the duck 220 was or was not shot. Alternatively, the outcome value &bgr; can be one of a range of finite integers or real numbers, or one of a range of continuous values. In these cases, the extent to which a shot misses or hits the duck 220 is relevant. Thus, the closer the gun 225 comes to shooting the duck 220, the less the outcome value &bgr; is, and thus, a near miss will result in a relatively low outcome value &bgr;, whereas a far miss will result in a relatively high outcome value &bgr;. Of course, alternatively, the closer the gun 225 comes to shooting the duck 220, the greater the outcome value &bgr; is. What is significant is that the outcome value &bgr; correctly indicates the extent to which the shot misses the duck 220. More alternatively, the extent to which a shot hits the duck 220 is relevant. Thus, the less damage the duck 220 incurs, the less the outcome value &bgr; is, and the more damage the duck 220 incurs, the greater the outcome value &bgr; is.
[0338] The probability update module 320 is configured to receive the outcome value &bgr; from the outcome evaluation module 330 and output an updated game strategy (represented by game move probability distribution p) that the duck 220 will use to counteract the player's 215 strategy in the future. In the preferred embodiment, the probability update module 320 utilizes a linear reward-penalty P-type update. As an example, given a selection of the seventeen different moves 255, assume that the gun 125 fails to shoot the duck 120 after it takes game move &agr;3, thus creating an outcome value &bgr;=1. In this case, general updating equations [6] and [7] can be expanded using equations [10] and [11], as follows: 11 p 3 ( k + 1 ) = p 3 ( k ) + ∑ j = 1 j ≠ 3 17 ap j ( k ) ;
p1(k+1)=p1(k)−ap1(k);
p2(k+1)=p2(k)−ap2(k);
p4(k+1)=p4(k)−ap4(k);
p17(k+1)=p17(k)−ap17(k)
[0339] Thus, since the game move &agr;3 resulted in a successful outcome, the corresponding probability value p3 is increased, and the game move probability values px corresponding to the remaining game moves &agr;x are decreased.
[0340] If, on the other hand, the gun 125 shoots the duck 120 after it takes game move &agr;3, thus creating an outcome value &bgr;=0, general updating equations [8] and [9] can be expanded, using equations [10] and [11], as follows: 12 p 3 ( k + 1 ) = p 3 ( k ) - ∑ j = 1 j ≠ 3 17 ( b 16 - bp j ( k ) ) p 1 ( k + 1 ) = p 1 ( k ) + b 16 - bp 1 ( k ) ; p 2 ( k + 1 ) = p 2 ( k ) + b 16 - bp 2 ( k ) ; p 4 ( k + 1 ) = p 4 ( k ) + b 16 - bp 4 ( k ) ; ⋮ p 17 ( k + 1 ) = p 17 ( k ) + b 16 - bp 17 ( k )
[0341] It should be noted that in the case where the gun 125 shoots the duck 120, thus creating an outcome value &bgr;=0, rather than using equations [8], [9], and [11], a value proportional to the penalty parameter b can simply be subtracted from the selection game move, and can then be equally distributed among the remaining game moves &agr;j. It has been empirically found that this method ensures that no probability value px converges to “1,” which would adversely result in the selection of a single game move &agr;x every time. In this case, equations [8] and [9] can be modified to read:
px(k+1)=px(k)−bpx(k) [8b]
[0342] 13 p j ( k + 1 ) = p j ( k ) + 1 n - 1 bp i ( k ) [9b]
[0343] Assuming game move &agr;3 results in an outcome value &bgr;=0, equations [8b] and [9b] can be expanded as follows:
p3(k+1)=p3(k)−bp3(k) 14 p 1 ( k + 1 ) = p 1 ( k ) + b 16 p 1 ( k ) ; p 2 ( k + 1 ) = p 2 ( k ) + b 16 p 2 ( k ) ; p 4 ( k + 1 ) = p 4 ( k ) + b 16 p 4 ( k ) ; ⋮ p 17 ( k + 1 ) = p 17 ( k ) + b 16 p 17 ( k )
[0344] In any event, since the game move a3 resulted in an unsuccessful outcome, the corresponding probability value p3 is decreased, and the game move probability values px corresponding to the remaining game moves &agr;j are increased. The values of a and b are selected based on the desired speed and accuracy that the learning module 310 learns, which may depend on the size of the game move set &agr;. For example, if the game move set &agr; is relatively small, the game 200 preferably must learn quickly, thus translating to relatively high a and b values. On the contrary, if the game move set a is relatively large, the game 200 preferably learns more accurately, thus translating to relatively low a and b values. In other words, the greater the values selected for a and b, the faster the game move probability distribution p changes, whereas the lesser the values selected for a and b, the slower the game move probability distribution p changes. In the preferred embodiment, the values of a and b have been chosen to be 0.1 and 0.5, respectively.
[0345] In the preferred embodiment, the reward-penalty update scheme allows the skill level of the game 200 to track that of the player 215 during gradual changes in the player's 215 skill level. Alternatively, a reward-inaction update scheme can be employed to constantly make the game 200 more difficult, e.g., if the game 200 has a training mode to train the player 215 to become progressively more skillful. More alternatively, a penalty-inaction update scheme can be employed, e.g., to quickly reduce the skill level of the game 200 if a different less skillful player 215 plays the game 200. In any event, the intuition module 315 may operate on the probability update module 320 to dynamically select any one of these update schemes depending on the objective to be achieved.
[0346] It should be noted that rather than, or in addition to, modifying the functionality of the game move selection module 325 by subdividing the game move set a into a plurality of game move subsets &agr;s, the respective skill levels of the game 200 and player 215 can be continuously and dynamically matched by modifying the functionality of the probability update module 320 by modifying or selecting the algorithms employed by it. For example, the respective reward and penalty parameters a and b may be dynamically modified.
[0347] For example, if the difference between the respective player and game scores 260 and 265 (i.e., the score difference value &Dgr;) is substantially positive, the respective reward and penalty parameters a and b can be increased, so that the skill level of the game 200 more rapidly increases. That is, if the gun 125 shoots the duck 120 after it takes a particular game move &agr;x, thus producing an unsuccessful outcome, an increase in the penalty parameter b will correspondingly decrease the chances that the particular game move &agr;x is selected again relative to the chances that it would have been selected again if the penalty parameter b had not been modified. If the gun 125 fails to shoot the duck 120 after it takes a particular game move &agr;x, thus producing a successful outcome, an increase in the reward parameter a will correspondingly increase the chances that the particular game move &agr;x is selected again relative to the chances that it would have been selected again if the penalty parameter a had not been modified. Thus, in this scenario, the game 200 will learn at a quicker rate.
[0348] On the contrary, if the score difference value &Dgr; is substantially negative, the respective reward and penalty parameters a and b can be decreased, so that the skill level of the game 200 less rapidly increases. That is, if the gun 125 shoots the duck 120 after it takes a particular game move &agr;x, thus producing an unsuccessful outcome, a decrease in the penalty parameter b will correspondingly increase the chances that the particular game move &agr;x is selected again relative to the chances that it would have been selected again if the penalty parameter b had not been modified. If the gun 125 fails to shoot the duck 120 after it takes a particular game move &agr;x, thus producing a successful outcome, a decrease in the reward parameter a will correspondingly decrease the chances that the particular game move &agr;i is selected again relative to the chances that it would have been selected again if the reward parameter a had not been modified. Thus, in this scenario, the game 200 will learn at a slower rate.
[0349] If the score difference value &Dgr; is low, whether positive or negative, the respective reward and penalty parameters a and b can remain unchanged, so that the skill level of the game 200 will tend to remain the same. Thus, in this scenario, the game 200 will learn at the same rate.
[0350] It should be noted that an increase or decrease in the reward and penalty parameters a and b can be effected in various ways. For example, the values of the reward and penalty parameters a and b can be incrementally increased or decreased a fixed amount, e.g., 0.1. Or the reward and penalty parameters a and b can be expressed in the functional form y=f(x), with the performance index &phgr; being one of the independent variables, and the penalty and reward parameters a and b being at least one of the dependent variables. In this manner, there is a smoother and continuous transition in the reward and penalty parameters a and b.
[0351] Optionally, to further ensure that the skill level of the game 200 rapidly decreases when the score difference value &Dgr; substantially negative, the respective reward and penalty parameters a and b can be made negative. That is, if the gun 125 shoots the duck 120 after it takes a particular game move &agr;x, thus producing an unsuccessful outcome, forcing the penalty parameter b to a negative number will increase the chances that the particular game move &agr;x is selected again in the absolute sense. If the gun 125 fails to shoot the duck 120 after it takes a particular game move &agr;x, thus producing a successful outcome, forcing the reward parameter a to a negative number will decrease the chances that the particular game move &agr;x is selected again in the absolute sense. Thus, in this scenario, rather than learn at a slower rate, the game 200 will actually unlearn. It should be noted in the case where negative probability values px result, the probability distribution p is preferably normalized to keep the game move probability values p, within the [0,1] range.
[0352] More optionally, to ensure that the skill level of the game 200 substantially decreases when the score difference value &Dgr; is substantially negative, the respective reward and penalty equations can be switched. That is, the reward equations, in this case equations [6] and [7], can be used when there is an unsuccessful outcome (i.e., the gun 125 shoots the duck 120). The penalty equations, in this case equations [8] and [9] (or [8b] and [9b]), can be used when there is a successful outcome (i.e., when the gun 125 misses the duck 120). Thus, the probability update module 320 will treat the previously selected &agr;i as producing an unsuccessful outcome, when in fact, it has produced a successful outcome, and will treat the previously selected &agr;x as producing a successful outcome, when in fact, it has produced an unsuccessful outcome. In this case, when the score difference value &Dgr; is substantially negative, the respective reward and penalty parameters a and b can be increased, so that the skill level of the game 200 more rapidly decreases.
[0353] Alternatively, rather than actually switching the penalty and reward equations, the functionality of the outcome evaluation module 330 can be modified with similar results. For example, the outcome evaluation module 330 may be modified to output an outcome value &bgr;=0 when the current game move a is successful, i.e., the gun 125 does not shoot the duck 120, and to output an outcome value &bgr;=1 when the current game move &agr;x is unsuccessful, i.e., the gun 125 shoots the duck 120. Thus, the probability update module 320 will interpret the outcome value &bgr; as an indication of an unsuccessful outcome, when in fact, it is an indication of a successful outcome, and will interpret the outcome value &bgr; as an indication of a successful outcome, when in fact, it is an indication of an unsuccessful outcome. In this manner, the reward and penalty equations are effectively switched.
[0354] Rather than modifying or switching the algorithms used by the probability update module 320, the game move probability distributions can be transformed. For example, if the score difference value &Dgr; is substantially positive, it is assumed that the game moves &agr;x corresponding to a set of the highest probability values px are too easy, and the game moves &agr;x corresponding to a set of the lowest probability values px are too hard. In this case, the game moves &agr;x corresponding to the set of highest probability values px can be switched with the game moves corresponding to the set of lowest probability values px, thereby increasing the chances that that the harder game moves &agr;x (and decreasing the chances that the easier game moves &agr;x) are selected relative to the chances that they would have been selected again if the game move probability distribution p had not been transformed. Thus, in this scenario, the game 200 will learn at a quicker rate. In contrast, if the score difference value &Dgr; is substantially negative, it is assumed that the game moves &agr;x corresponding to the set of highest probability values px are too hard, and the game moves &agr;x corresponding to the set of lowest probability values px are too easy. In this case, the game moves &agr;x corresponding to the set of highest probability values px can be switched with the game moves corresponding to the set of lowest probability values px, thereby increasing the chances that that the easier game moves &agr;x (and decreasing the chances that the harder game moves &agr;x) are selected relative to the chances that they would have been selected again if the game move probability distribution p had not been transformed. Thus, in this scenario, the game 200 will learn at a slower rate. If the score difference value &Dgr; is low, whether positive or negative, it is assumed that the game moves &agr;x corresponding to the set of highest probability values px are not too hard, and the game moves &agr;x corresponding to the set of lowest probability values px are not too easy, in which case, the game moves &agr;x corresponding to the set of highest probability values px and set of lowest probability values px are not switched. Thus, in this scenario, the game 200 will learn at the same rate.
[0355] It should be noted that although the performance index &phgr; has been described as being derived from the score difference value &Dgr;, the performance index &phgr; can also be derived from other sources, such as the game move probability distribution p. If it is known that the outer moves 255a or more difficult than the inner moves 255b, the performance index &phgr;, and in this case, the skill level of the player 215 relative to the skill level the game 200, may be found in the present state of the game move probability values px assigned to the moves 255. For example, if the combined probability values px corresponding to the outer moves 255a is above a particular threshold value, e.g., 0.7 (or alternatively, the combined probability values p, corresponding to the inner moves 255b is below a particular threshold value, e.g., 0.3), this may be an indication that the skill level of the player 215 is substantially greater than the skill level of the game 200. In contrast, if the combined probability values px corresponding to the outer moves 255a is below a particular threshold value, e.g., 0.4 (or alternatively, the combined probability values px corresponding to the inner moves 255b is above a particular threshold value, e.g., 0.6), this may be an indication that the skill level of the player 215 is substantially less than the skill level of the game 200. Similarly, if the combined probability values px corresponding to the outer moves 255a is within a particular threshold range, e.g., 0.4-0.7 (or alternatively, the combined probability values px corresponding to the inner moves 255b is within a particular threshold range, e.g., 0.3-0.6), this may be an indication that the skill level of the player 215 and skill level of the game 200 are substantially matched. In this case, any of the afore-described probabilistic learning module modification techniques can be used with this performance index &phgr;.
[0356] Alternatively, the probabilities values px corresponding to one or more game moves &agr;x can be limited to match the respective skill levels of the player 215 and game 200. For example, if a particular probability value px is too high, it is assumed that the corresponding game move &agr;x may be too hard for the player 215. In this case, one or more probabilities values px can be limited to a high value, e.g., 0.4, such that when a probability value px reaches this number, the chances that that the corresponding game move &agr;x is selected again will decrease relative to the chances that it would be selected if the corresponding game move probability px had not been limited. Similarly, one or more probabilities values p, can be limited to a low value, e.g., 0.01, such that when a probability value px reaches this number, the chances that that the corresponding game move &agr;x is selected again will increase relative to the chances that it would be selected if the corresponding game move probability px had not been limited. It should be noted that the limits can be fixed, in which case, only the performance index &phgr; that is a function of the game move probability distribution p is used to match the respective skill levels of the player 215 and game 200, or the limits can vary, in which case, such variance may be based on a performance index &phgr; external to the game move probability distribution p.
[0357] Having now described the structure of the game program 300, the steps performed by the game program 300 will be described with reference to FIG. 9. First, the game move probability distributions is initialized (step 405). Specifically, the probability update module 320 initially assigns an equal probability value to each of the game moves &agr;x, in which case, the initial game move probability distribution p(k) can be represented by 15 p 1 ( 0 ) = p 2 ( 0 ) = p 2 ( 0 ) = … p n ( 0 ) = 1 n .
[0358] Thus, all of the game moves &agr;x have an equal chance of being selected by the game move selection module 325. Alternatively, probability update module 320 initially assigns unequal probability values to at least some of the game moves &agr;x For example, the outer moves 255a may be initially assigned a lower probability value than that of the inner moves 255b, so that the selection of any of the outer moves 255a as the next game move &agr;x will be decreased. In this case, the duck 220 will not be too difficult to shoot when the game 200 is started. In addition to the game move probability distribution p, the current game move &agr;x to be updated is also initialized by the probability update module 320 at step 405.
[0359] Then, the game move selection module 325 determines whether a player move &lgr;2x has been performed, and specifically whether the gun 225 has been fired by clicking the mouse button 245 (step 410). If a player move &lgr;2x has been performed, the outcome evaluation module 330 determines whether the last game move &agr;x was successful by performing a collision detection, and then generates the outcome value &bgr; in response thereto (step 415). The intuition module 315 then updates the player score 260 and duck score 265 based on the outcome value &bgr; (step 420). The probability update module 320 then, using any of the updating techniques described herein, updates the game move probability distribution p based on the generated outcome value &bgr; (step 425).
[0360] After step 425, or if a player move &lgr;2x has not been performed at step 410, the game move selection module 325 determines if a player move &lgr;1x has been performed, i.e., gun 225, has breached the gun detection region 270 (step 430). If the gun 225 has not breached the gun detection region 270, the game move selection module 325 does not select any game move &agr;x from the game move subset &agr; and the duck 220 remains in the same location (step 435). Alternatively, the game move &agr;x may be randomly selected, allowing the duck 220 to dynamically wander. The game program 300 then returns to step 410 where it is again determined if a player move &lgr;2x has been performed. If the gun 225 has breached the gun detection region 270 at step 430, the intuition module 315 modifies the functionality of the game move selection module 325 based on the performance index &phgr;, and the game move selection module 325 selects a game move &agr;x from the game move set a.
[0361] Specifically, the intuition module 315 determines the relative player skill level by calculating the score difference value &Dgr; between the player score 260 and duck score 265 (step 440). The intuition module 315 then determines whether the score difference value &Dgr; is greater than the upper score difference threshold NS2 (step 445). If A is greater than NS2, the intuition module 315, using any of the game move subset selection techniques described herein, selects a game move subset &agr;S, a corresponding average probability of which is relatively high (step 450). If &Dgr; is not greater than NS2, the intuition module 315 then determines whether the score difference value &Dgr; is less than the lower score difference threshold NS1 (step 455). If &Dgr; is less than NS1, the intuition module 315, using any of the game move subset selection techniques described herein, selects a game move subset &agr;s, a corresponding average probability of which is relatively low (step 460). If &Dgr; is not less than NS1, it is assumed that the score difference value &Dgr; is between NS1 and NS2, in which case, the intuition module 315, using any of the game move subset selection techniques described herein, selects a game move subset &agr;s, a corresponding average probability of which is relatively medial (step 465). In any event, the game move selection module 325 then pseudo-randomly selects a game move &agr;x from the selected game move subset &agr;s, and accordingly moves the duck 220 in accordance with the selected game move &agr;x (step 470). The game program 300 then returns to step 410, where it is determined again if a player move &lgr;2x has been performed.
[0362] It should be noted that, rather than use the game move subset selection technique, the other afore-described techniques used to dynamically and continuously match the skill level of the player 215 with the skill level of the game 200 can be alternatively or optionally be used as well. For example, and referring to FIG. 10, the probability update module 320 initializes the game move probability distribution p and current game move &agr;x similarly to that described in step 405 of FIG. 9. The initialization of the game move probability distribution p and current game move &agr;x is similar to that performed in step 405 of FIG. 9. Then, the game move selection module 325 determines whether a player move &lgr;2x has been performed, and specifically whether the gun 225 has been fired by clicking the mouse button 245 (step 510). If a player move &lgr;2x has been performed, the intuition module 315 modifies the functionality of the probability update module 320 based on the performance index &phgr;.
[0363] Specifically, the intuition module 315 determines the relative player skill level by calculating the score difference value &Dgr; between the player score 260 and duck score 265 (step 515). The intuition module 315 then determines whether the score difference value &Dgr; is greater than the upper score difference threshold NS2 (step 520). If A is greater than NS2, the intuition module 315 modifies the functionality of the probability update module 320 to increase the game's 200 rate of learning using any of the techniques described herein (step 525). For example, the intuition module 315 may modify the parameters of the learning algorithms, and specifically, increase the reward and penalty parameters a and b.
[0364] If &Dgr; is not greater than NS2, the intuition module 315 then determines whether the score difference value &Dgr; is less than the lower score difference threshold NS1 (step 530). If &Dgr; is less than NS1, the intuition module 315 modifies the functionality of the probability update module 320 to decrease the game's 200 rate of learning (or even make the game 200 unlearn) using any of the techniques described herein (step 535). For example, the intuition module 315 may modify the parameters of the learning algorithms, and specifically, decrease the reward and penalty parameters a and b. Alternatively or optionally, the intuition module 315 may assign the reward and penalty parameters a and b negative numbers, switch the reward and penalty learning algorithms, or even modify the outcome evaluation module 330 to output an outcome value &bgr;=0 when the selected game move &agr;x is actually successful, and output an outcome value &bgr;=1 when the selected game move &agr;i is actually unsuccessful.
[0365] If &Dgr; is not less than NS2, it is assumed that the score difference value &Dgr; is between NS1 and NS2, in which case, the intuition module 315 does not modify the probability update module 320 (step 540).
[0366] In any event, the outcome evaluation module 330 then determines whether the last game move &agr;x was successful by performing a collision detection, and then generates the outcome value &agr;x in response thereto (step 545). Of course, if the intuition module 315 modifies the functionality of the outcome evaluation module 330 during any of the steps 525 and 535, step 545 will preferably be performed during these steps. The intuition module 315 then updates the player score 260 and duck score 265 based on the outcome value &bgr; (step 550). The probability update module 320 then, using any of the updating techniques described herein, updates the game move probability distributionp based on the generated outcome value &bgr; (step 555).
[0367] After step 555, or if a player move &lgr;2x has not been performed at step 510, the game move selection module 325 determines if a player move &lgr;1x has been performed, i.e., gun 225, has breached the gun detection region 270 (step 560). If the gun 225 has not breached the gun detection region 270, the game move selection module 325 does not select a game move &agr;x from the game move set &agr; and the duck 220 remains in the same location (step 565). Alternatively, the game move &agr;x may be randomly selected, allowing the duck 220 to dynamically wander. The game program 300 then returns to step 510 where it is again determined if a player move &lgr;2x has been performed. If the gun 225 has breached the gun detection region 270 at step 560, the game move selection module 325 pseudo-randomly selects a game move &agr;x from the game move set &agr; and accordingly moves the duck 220 in accordance with the selected game move &agr;x (step 570). The game program 300 then returns to step 510, where it is determined again if a player move &lgr;2x has been performed.
[0368] More specific details on the above-described operation of the duck game 100 can be found in the Computer Program Listing Appendix attached hereto and previously incorporated herein by reference. It is noted that each of the files “Intuition Intelligence-duckgame1.doc” and “Intuition Intelligence-duckgame2.doc” represents the game program 300, with file “Intuition Intelligence-duckgame1.doc” utilizing the game move subset selection technique to continuously and dynamically match the respective skill levels of the game 200 and player 215, and file “Intuition “Intuition Intelligence-duckgame2.doc” utilizing the learning algorithm modification technique (specifically, modifying the respective reward and penalty parameters a and b when the score difference value &Dgr; is too positive or too negative, and switching the respective reward and penalty equations when the score difference value &Dgr; is too negative) to similarly continuously and dynamically match the respective skill levels of the game 200 and player 215.
[0369] Single-Player Educational Program (Single Game Move-Single Player Move)
[0370] The learning program 100 can be applied to other applications besides game programs. A single-player educational program 700 (shown in FIG. 12) developed in accordance with the present inventions is described in the context of a child's learning toy 600 (shown in FIG. 11), and specifically, a doll 600 and associated articles of clothing and accessories 610 that are applied to the doll 600 by a child 605 (shown in FIG. 12). In the illustrated embodiment, the articles 610 include a (1) purse, calculator, and hairbrush, one of which can be applied to a hand 615 of the doll 600; (2) shorts and pants, one of which can be applied to a waist 620 of the doll 600; (3) shirt and tank top, one of which can be applied to a chest 625 of the doll 600; and (4) dress and overalls, one of which can be applied to the chest 625 of the doll 600. Notably, the dress and overalls cover the waist 620, so that the shorts and pants cannot be applied to the doll 600 when the dress or overalls are applied. Depending on the measured skill level of the child 605, the doll 600 will instruct the child 605 to apply either a single article, two articles, or three articles to the doll 600. For example, the doll 600 may say “Simon says, give me my calculator, pants, and tank top.”
[0371] In accordance with the instructions given by the doll 600, the child 605 will then attempt to apply the correct articles 610 to the doll 600. For example, the child 605 may place the calculator in the hand 615, the pants on the waist 620, and the tank top on the chest 625. To determine which articles 610 the child 605 has applied, the doll 600 comprises sensors 630 located on the hand 615, waist 620, and chest 625. These sensors 630 sense the unique resistance values exhibited by the articles 610, so that the doll 600 can determine which of the articles 610 are being applied.
[0372] As illustrated in Tables 2-4, there are 43 combinations of articles 610 that can be applied to the doll 600. Specifically, actions &agr;1-&agr;9 represent all of the single article combinations, actions &agr;10-&agr;31 represent all of the double article combinations, and actions &agr;32-&agr;43 represent all of the triple article combinations that can be possibly applied to the doll 600. 2 TABLE 2 Exemplary Single Article Combinations for Doll Action (&agr;) Hand Waist Chest &agr;1 Purse x x &agr;2 Calculator x x &agr;3 Hairbrush x x &agr;4 x Shorts x &agr;5 x Pants x &agr;6 x x Shirt &agr;7 x x Tanktop &agr;8 x x Dress &agr;9 x x Overalls
[0373] 3 TABLE 3 Exemplary Double article combinations for Doll Action (&agr;) Hand Waist Chest &agr;10 Purse Shorts x &agr;11 Purse Pants x &agr;12 Purse x Shirt &agr;13 Purse x Tanktop &agr;14 Purse x Dress &agr;15 Purse x Overalls &agr;16 Calculator Shorts x &agr;17 Calculator Pants x &agr;18 Calculator x Shirt &agr;19 Calculator x Tanktop &agr;20 Calculator x Dress &agr;21 Calculator x Overalls &agr;22 Hairbrush Shorts x &agr;23 Hairbrush Pants x &agr;24 Hairbrush x Shirt &agr;25 Hairbrush x Tanktop &agr;26 Hairbrush x Dress &agr;27 Hairbrush x Overalls &agr;28 x Shorts Shirt &agr;29 x Shorts Tanktop &agr;30 x Pants Shirt &agr;31 x Pants Tanktop
[0374] 4 TABLE 4 Exemplary Three Article Combinations for Doll Action (&agr;) Hand Waist Chest &agr;32 Purse Shorts Shirt &agr;33 Purse Shorts Tanktop &agr;34 Purse Pants Shirt &agr;35 Purse Pants Tanktop &agr;36 Calculator Shorts Shirt &agr;37 Calculator Shorts Tanktop &agr;38 Calculator Pants Shirt &agr;39 Calculator Pants Tanktop &agr;40 Hairbrush Shorts Shirt &agr;41 Hairbrush Shorts Tanktop &agr;42 Hairbrush Pants Shirt &agr;43 Hairbrush Pants Tanktop
[0375] In response to the selection of one of these actions &agr;x, i.e., prompting the child 605 to apply one of the 43 article combinations to the doll 600, the child 605 will attempt to apply the correct article combinations to the doll 600, represented by corresponding child actions &lgr;1-&lgr;43. It can be appreciated an article combination &lgr;x will be correct if it corresponds to the article combination &agr;x prompted by the doll 600 (i.e., the child action &lgr; corresponds with the doll action &agr;), and will be incorrect if it corresponds to the article combination &agr;x prompted by the doll 600 (i.e., the child action &lgr; does not correspond with the doll action &Dgr;).
[0376] The doll 600 seeks to challenge the child 605 by prompting him or her with more difficult article combinations as the child 605 applies correct combinations to the doll 600. For example, if the child 605 exhibits a proficiency at single article combinations, the doll 600 will prompt the child 605 with less single article combinations and more double and triple article combinations. If the child 605 exhibits a proficiency at double article combinations, the doll 600 will prompt the child 605 with less single and double article combinations and more triple article combinations. If the child 605 exhibits a proficiency at three article combinations, the doll 600 will prompt the child 605 with even more triple article combinations.
[0377] The doll 600 also seeks to avoid over challenging the child 605 and frustrating the learning process. For example, if the child 605 does not exhibit a proficiency at triple article combinations, the doll 600 will prompt the child 605 with less triple article combinations and more single and double article combinations. If the child 605 does not exhibit a proficiency at double article combinations, the doll 600 will prompt the child 605 with less double and triple article combinations and more single article combinations. If the child 605 does not exhibit a proficiency at single article combinations, the doll 600 will prompt the child 605 with even more single article combinations.
[0378] To this end, the educational program 700 generally includes a probabilistic learning module 710 and an intuition module 715, which are specifically tailored for the doll 600. The probabilistic learning module 710 comprises a probability update module 720, an article selection module 725, and an outcome evaluation module 730. Specifically, the probability update module 720 is mainly responsible for learning the child's current skill level, with the outcome evaluation module 730 being responsible for evaluating the article combinations &agr;x prompted by the doll 600 relative to the article combinations &lgr;x, selected by the child 605. The article selection module 725 is mainly responsible for using the learned skill level of the child 605 to select the article combinations &agr;x that are used to prompt the child 605. The intuition module 715 is responsible for directing the learning of the educational program 700 towards the objective, and specifically, dynamically pushing the skill level of the child 605 to a higher level. In this case, the intuition module 715 operates on the probability update module 720, and specifically selects the methodology that the probability update module 720 will use to update an article probability distribution p.
[0379] To this end, the outcome evaluation module 730 is configured to receive an article combination &agr;x from the article selection module 725 (i.e., one of the forty-three article combinations that are prompted by the doll 600), and receive an article combination &lgr;x from the child 605 (i.e., one of the forty-three article combinations that can be applied to the doll 600). The outcome evaluation module 730 is also configured to determine whether each article combination &lgr;x received from the child 605 matches the article combination &agr;x prompted by the doll 600, with the outcome value &bgr; equaling one of two predetermined values, e.g., “0” if there is a match and “1” if there is not a match. In this case, a P-type learning methodology is used. Optionally, the outcome evaluation module 730 can generate an outcome value &bgr; equaling a value between “0” and “1.” For example, if the child 605 is relatively successful by matching most of the articles within the prompted article combination &agr;x, the outcome value &bgr; can be a lower value, and if the child 605 is relatively unsuccessful by not matching most of the articles within the prompted article combination &agr;x, the outcome value &bgr; can be a higher value. In this case, Q- and S-type learning methodologies can be used. In contrast to the duck game 200 where the outcome value &bgr; measured the success or failure of a duck move relative to the game player, the performance of a prompted article combination &agr;x is not characterized as being successful or unsuccessful, since the doll 600 is not competing with the child 605, but rather serves to teach the child 605.
[0380] The probability update module 720 is configured to generate and update the article probability distribution p in a manner directed by the intuition module 715, with the article probability distribution p containing forty-three probability values px corresponding to the forty-three article combinations &agr;x. In the illustrated embodiment, the forty-three article combinations &agr;i are divided amongst three article combination subsets &agr;s: &agr;s1 for the nine single article combinations; &agr;s2 for the twenty-two double article combinations; and &agr;s3 for the twelve three article combinations. When updating the article probability distribution p, the three article combination subsets &agr;s are updated as three actions, with the effects of each updated article combination subset &agr;s being evenly distributed amongst the article combinations &agr;s in the respective subset &agr;s. For example, if the single article combination subset &agr;s1 is increased by ten percent, each of the single article combinations &agr;1-&agr;9 will be correspondingly increased by ten percent.
[0381] The article selection module 725 is configured for receiving the article probability distribution p from the probability update module 720, and pseudo-randomly selecting the article combination &agr;x therefrom in the same manner as the game move selection module 325 of the program 300 selects a game move &agr;x from a selected game move subset &agr;s. Specifically, pseudo-random selection can be accomplished by first generating a progressive sum of the probability values px For example, Table 5 sets forth exemplary normalized probability values and a progressive sum for the forty-three article combinations &agr;x of the article probability distribution p: 5 TABLE 5 Progressive Sum of Probability Values For Forty-Three Exemplary Article Combinations Normalized Progressive Game Move Probability Value Sum &agr;1 0.059 0.059 &agr;2 0.059 0.118 &agr;3 0.059 0.187 . . . . . . . . . &agr;9 0.059 0.531 &agr;10 0.014 0.545 &agr;11 0.014 0.559 &agr;12 0.014 0.573 . . . . . . . . . &agr;31 0.014 0.839 &agr;32 0.013 0.852 &agr;33 0.013 0.865 . . . . . . . . . &agr;43 0.013 1.000
[0382] The article selection module 725 then selects a random number between “0” and “1,” and selects the article combination &agr;x corresponding to the next highest progressive sum value. For example, if the randomly selected number is 0.562, article combination &agr;11 (i.e., purse and pants) will be selected.
[0383] In an alternative embodiment, the article probability distribution p contains three probability values px respectively corresponding to three article combination subsets &agr;s, one of which can then be pseudo-randomly selected therefrom. In a sense, the article combination subsets &agr;s are treated as actions to be selected. For example, Table 6 sets forth exemplary normalized probability values and a progressive sum for the three article combination subsets &agr;s of the article probability distribution p: 6 TABLE 6 Progressive Sum of Probability Values For Three Exemplary Article Combination Subsets Game Move Normalized Probability Value Progressive Sum &agr;1 0.65 0.65 &agr;2 0.25 0.90 &agr;3 0.10 1.00
[0384] The article selection module 725 then selects a random number between “0” and “1,” and selects the article combination subset &agr;s corresponding to the next highest progressive sum value. For example, if the randomly selected number is 0.78, article combination subset &agr;s2 will be selected. After the article combination subset &agr;s has been pseudo-randomly selected, the article selection module 725 then randomly selects an article combination &agr;x from that selected combination subset &agr;s. For example, if the second article combination subset &agr;s was selected, the article selection module 725 will randomly select one of the twenty-two double article combinations &agr;10-&agr;31.
[0385] The intuition module 715 is configured to modify the functionality of the probability update module 720 based on the performance index &phgr;, and in this case, the current skill level of the child 605 relative to the current teaching level of the doll 600. In the preferred embodiment, the performance index &phgr; is quantified in terms of the degree of difficulty of the currently prompted article combination &agr;x and the outcome value &bgr; (i.e., whether or not the child 605 successfully matched the article combination &agr;x). In this respect, the performance index &phgr; is instantaneous. It should be appreciated, however, that the performance of the educational program 700 can also be based on a cumulative performance index &phgr;. For example, the educational program 700 can keep track of a percentage of the child's matching article combinations &lgr;x broken down by difficulty level of the prompted article combinations &agr;x.
[0386] It can be appreciated, that applying only one article to the doll 600 is an easier task than applying two articles to the doll 600, which is in turn an easier task then applying three articles to the doll 600 in an given time. The intuition module 715 will attempt to “push” the child's skill level higher, so that the child 605 will consistently be able to correctly apply two articles, and then three articles 610, to the doll 600.
[0387] The intuition module 715 modifies the functionality of the probability update module 720 by determining which updating methodology will be used. The intuition module 715 also determines which article combination a will be rewarded or penalized, which is not necessarily the article combination that was previously selected by the article selection module 725.
[0388] Referring to FIGS. 13a-f, various methodologies can be selected by the intuition module 715 to update the article probability distribution p, given a currently prompted article combination &agr;x and outcome value &bgr;. Although, the probability values px in the article probability distribution p have been described as corresponding to the individual article combinations &agr;, for purposes of simplicity and brevity, the probability values px depicted in FIGS. 13a-f respectively correspond with the single, double and triple article combination subsets &agr;s. As will be described in further detail below, the intuition module 715 directs the probability update module 720 to shift the article probability distribution p from probability value(s) p, corresponding to article combinations &agr;x associated with lesser difficult levels to probability value(s) px corresponding to article combinations &agr;x associated with greater difficult levels when the child 605 is relatively successful at matching the prompted article combination &agr;x, and to shift the article probability distribution p from probability value(s) px corresponding to article combinations &agr;x associated with greater difficult levels to probability value(s) px corresponding to article combinations &pgr;x associated with lesser difficult levels when the child 605 is relatively unsuccessful at matching the prompted article combination &agr;x. In the illustrated embodiment, P-type learning methodologies (&bgr; equals either “0” or “1”) are used, in which case, it is assumed that the child 605 is absolutely successful or unsuccessful at matching the prompted article combination &agr;x. Alternatively, Q- and S-type learning methodologies (&bgr; is between “0” and “1”) are used, in which case, it is assumed that the child 605 can partially match or not match the prompted article combination &agr;x. For example, the outcome value &bgr; may be a lesser value if the child 605 matches most of the articles in the prompted article combination &agr;x, (relatively successful), and may be a greater value if the child 605 does not match most of the articles in the prompted article combination &agr;x, (relatively unsuccessful).
[0389] FIG. 13a illustrates a methodology used to update the article probability distribution p when a single article combination subset &agr;s1 is currently selected, and the child 605 succeeds in matching the prompted article combination &agr; (i.e., &bgr;=0). In this case, the intuition module 715 will attempt to drive the child's skill level from the single article combination subset &agr;s1 to the double article combination subset &agr;s2 by increasing the probability that the child 605 will subsequently be prompted by the more difficult double subset combination sets &agr;s2 and triple subset combination sets &agr;s3. The intuition module 715 accomplishes this be shifting the probability distribution p from the probability value p1 to the probability values p2 and p3. Specifically, the single article combination subset &agr;s1 is penalized by subtracting a proportionate value equal to “x” (e.g., ⅕ of p1) from probability value p1 and distributing it to the probability values p2 and p3.
[0390] Since the child's success with a single article combination set &agr;s1 indicates that the child 605 may be relatively proficient at double article combinations &agr;s2, but not necessarily the more difficult triple article combinations &agr;s3, the probability value p2 is increased more than the probability value p3 to ensure that the child's skill level is driven from the single article combination subset &agr;s1 to the double article combination subset &agr;s2, and not overdriven to the third article combination subset &agr;s3. For example, the proportions of “x” added to the probability values p2 and p3 can be ⅔ and ⅓, respectively. In effect, the learning process will be made smoother for the child 605. Notably, the methodology illustrated in FIG. 13a allows control over the relative amounts that are added to the probability values p2 and p3. That is, the amount added to the probability value p2 will always be greater than the amount added to the probability value p3 irrespective of the current magnitudes of the probability values p2 and p3, thereby ensuring a smooth learning process.
[0391] General equations [20] and [21a] can be used to implement the learning methodology illustrated in FIG. 13a. Given that h1(p(k))=(⅕)p1(k), d12={fraction (2/3)}, and d13=⅓, equations [20] and [21a] can be broken down into: 16 p 2 ( k + 1 ) = p 2 ( k ) + ( 2 3 ) ( 1 5 ) p 1 ( k ) = p 2 ( k ) + 2 15 p 1 ( k ) ; and [21a-1] p 3 ( k + 1 ) = p 3 ( k ) + ( 1 3 ) ( 1 5 ) p 1 ( k ) = p 3 ( k ) + 1 15 p 1 ( k ) [21a-2]
[0392] FIG. 13b illustrates a methodology used to update the article probability distribution p when a single article combination subset &agr;s1 is currently selected, and the child 605 does not succeed in matching the prompted article combination &agr; (i.e., &bgr;=1). In this case, the intuition module 715 will attempt to prevent over-challenging the child 605 by decreasing the probability that the child 605 will subsequently be prompted by the more difficult double and triple subset combination sets &agr;s2 and &agr;s3. The intuition module 715 accomplishes this be shifting the probability distribution p from the probability values p2 and p3 to the probability value p1. Specifically, the single article combination subset &agr;s1 is rewarded by subtracting a proportional value equal to “x” from probability value P2 and adding it to the probability value p1, and subtracting a proportionate value equal to “y” from probability value p3 and adding it to the probability value p1.
[0393] Since the child's failure with a single article combination set &agr;s1 indicates that the child 605 may not be proficient at double and triple article combinations &agr;s2 and &agr;s3, the intuition module 715 attempts to adapt to the child's apparently low skill level by decreasing the probability values p2 and p3 as quickly as possible. Because the probability value p2 will most likely be much greater than the probability value p3 if the child 605 is not proficient at the single article combination sets &agr;s2, the intuition module 715 adapts to the child's low skill level by requiring that the proportionate amount that is subtracted from the probability value p2 be greater than that subtracted from the probability value p3, i.e., the proportionate value “x” is set higher than the proportional value “y”. For example, “x” can equal {fraction (2/15)} and “y” can equal {fraction (1/15)}.
[0394] Notably, the methodology illustrated in FIG. 13b allows control over the proportionate amounts that are subtracted from the probability values p2 and p3 and added to the probability value p1, so that the doll 600 can quickly adapt to a child's lower skill level in a stable manner. That is, if the probability values p2 and p3 are relatively high, a proportionate amount subtracted from these probability values will quickly decrease them and increase the probability value pi, whereas if the probability values p2 and p3 are relatively low, a proportionate amount subtracted from these probability values will not completely deplete them.
[0395] General equations [6a]-[7a] can be used to implement the learning methodology illustrated in FIG. 13b. Given that g12(p(k))=({fraction (2/15)})p2(k) and g13(p(k))=({fraction (1/15)})p3(k), equations [6a]-[7a] can be broken down into: 17 p 1 ( k + 1 ) = p 1 ( k ) + ∑ j = 2 3 g 1 j ( p ( k ) ) = p 1 ( k ) + 2 15 p 2 ( k ) + 1 15 p 3 ( k ) ; [6a-1] p 2 ( k + 1 ) = p 2 ( k ) - g 12 ( p ( k ) ) = p 2 ( k ) - 2 15 p 2 ( k ) = 13 15 p 2 ( k ) ; and [7a-1] p 3 ( k + 1 ) = p 3 ( k ) - g 13 ( p ( k ) ) = p 3 ( k ) - 1 15 p 3 ( k ) = 14 15 p 3 ( k ) [7a-2]
[0396] FIG. 13c illustrates a methodology used to update the article probability distribution p when a double article combination subset &agr;s2 is currently selected, and the child 605 succeeds in matching the prompted article combination a (i.e., &bgr;=0). In this case, the intuition module 715 will attempt to drive the child's skill level from the double article combination subset a&agr;s2 to the triple article combination subset &agr;s3 by increasing the probability that the child 605 will subsequently be prompted by the more difficult triple subset combination sets &agr;s3. The intuition module 715 accomplishes this be shifting the probability distribution p from the probability value p to the probability values p2 and p3. Specifically, the single article combination subset &agr;s1 is penalized by subtracting a proportionate value equal to “x” (e.g., ⅕ of p1) from probability value p1 and distributing it to the probability values p2 and p3.
[0397] Since the child's success with a double article combination set &agr;s2 indicates that the child 605 may be relatively proficient at triple article combinations &agr;s2, the probability value p3 is increased more than the probability value p2 to ensure that the child's skill level is driven from the double article combination subset &agr;s2 to the triple article combination subset &agr;s3. For example, the proportions of “x” added to the probability values p2 and p3 can be ⅓ and ⅔, respectively. Notably, the methodology illustrated in FIG. 13c allows control over the relative amounts that are added to the probability values p2 and p3. That is, the amount added to the probability value p3 will always be greater than the amount added to the probability value p2 irrespective of the current magnitudes of the probability values p2 and p3, thereby ensuring that the child's skill level is driven towards the triple article combination subset &agr;s3, rather than maintaining the child's skill level at the double article combination subset &agr;s2.
[0398] General equations [20] and [21a] can be used to implement the learning methodology illustrated in FIG. 13c. Given that h1(p(k))=({fraction (1/15)})p1(k), d12={fraction (1/3)}, and d13=⅔, equations [20] and [21a] can be broken down into: 18 p 1 ( k + 1 ) = p 1 ( k ) - h 1 ( p ( k ) ) = p 1 ( k ) - 1 5 p 1 ( k ) = 4 5 p 1 ( k ) ; [20-2] p 2 ( k + 1 ) = p 2 ( k ) + ( 1 3 ) ( 1 5 ) p 1 ( k ) = p 2 ( k ) + 1 15 p 1 ( k ) ; and [21a-3] p 3 ( k + 1 ) = p 3 ( k ) + ( 2 3 ) ( 1 5 ) p 1 ( k ) = p 3 ( k ) + 2 15 p 1 ( k ) [21a-4]
[0399] FIG. 13d illustrates a methodology used to update the article probability distributions when a double article combination subset &agr;s2 is currently selected, and the child 605 does not succeed in matching the prompted article combination &agr; (i.e., &bgr;=1). In this case, the intuition module 715 will attempt to prevent over-challenging the child 605 by decreasing the probability that the child 605 will subsequently be prompted by the more difficult double and triple subset combination sets &agr;s2 and &agr;s3. The intuition module 715 accomplishes this by shifting the probability distribution p from the probability values p2 and p3 to the probability value p1. Specifically, the single article combination subset &agr;s1 is rewarded by subtracting a proportional value equal to “x” from probability value p2 and adding it to the probability value pi, and subtracting a proportionate value equal to “y” from probability value p3 and adding it to the probability value p1.
[0400] Since the child's failure with a double article combination set &agr;s2 indicates that the child 605 may not be proficient at triple article combinations &agr;s2, the probability value p3 is decreased more than the probability value p2. The intuition module 715 accomplishes this by requiring that the proportionate amount that is subtracted from the probability value p3 be greater than that subtracted from the probability value p2, i.e., the proportionate value “y” is set higher than the proportional value “x”. For example, “x” can equal {fraction (1/15)} and “y” can equal {fraction (2/15)}.
[0401] Notably, the methodology illustrated in FIG. 13d allows control over the proportionate amounts that are subtracted from the probability values p2 and p3 and added to the probability value p1, so that the doll 600 can quickly adapt to a child's lower skill level in a stable manner. That is, if the probability values p2 and p3 are relatively high, a proportionate amount subtracted from these probability values will quickly decrease them and increase the probability value p1, whereas if the probability values p2 and p3 are relatively low, a proportionate amount subtracted from these probability values will not completely deplete them.
[0402] General equations [6a]-[7a] can be used to implement the learning methodology illustrated in FIG. 13d. Given that g12(p(k))=({fraction (1/15)})p2(k) and g13(p(k))=({fraction (2/15)})p3(k), equations [6a]-[7a] can be broken down into: 19 p 1 ( k + 1 ) = p 1 ( k ) + ∑ j = 2 3 g 1 j ( p ( k ) ) = p 1 ( k ) + 1 15 p 2 ( k ) + 2 15 p 3 ( k ) ; [6a-2] p 2 ( k + 1 ) = p 2 ( k ) - g 12 ( p ( k ) ) = p 2 ( k ) - 1 15 p 2 ( k ) = 14 15 p 2 ( k ) ; and [7a-3] p 3 ( k + 1 ) = p 3 ( k ) - g 13 ( p ( k ) ) = p 3 ( k ) - 2 15 p 3 ( k ) = 13 15 p 3 ( k ) [7a-4]
[0403] FIG. 13e illustrates a methodology used to update the article probability distribution p when a triple article combination subset &agr;s3 is currently selected, and the child 605 succeeds in matching the prompted article combination &agr; (i.e., &bgr;=0). In this case, the intuition module 715 will attempt to drive the child's skill level further to the triple article combination subset &agr;s3 by increasing the probability that the child 605 will subsequently be prompted by the more difficult triple subset combination sets &agr;s3. The intuition module 715 accomplishes this be shifting the probability distribution p from the probability values p1 and p2 to the probability value p3 Specifically, the triple article combination subset &agr;s1 is rewarded by subtracting a proportional value equal to “x” from probability value p1 and adding it to the probability value p3, and subtracting a proportionate value equal to “y” from probability value p2 and adding it to the probability value p3.
[0404] Since the child 605 is much more proficient at single article combinations &agr;s1 than with double article combinations &agr;s2, the intuition module 715 attempts to reduce the probability value p1 more than the probability value p2. The intuition module 715 accomplishes this by requiring that the proportionate amount that is subtracted from the probability value p1 be greater than that subtracted from the probability value p2, i.e., the proportionate value “x” is set higher than the proportional value “y”. For example, “x” can equal {fraction (2/15)} and “y” can equal {fraction (1/15)}.
[0405] Notably, the methodology illustrated in FIG. 13e allows control over the proportionate amounts that are subtracted from the probability values p2 and p3 and added to the probability value p1, so that the doll 600 can quickly adapt to a child's higher skill level in a stable manner. That is, if the probability values p1 and p2 are relatively high, a proportionate amount subtracted from these probability values will quickly decrease them and increase the probability value p3, whereas if the probability values p1 and p2 are relatively low, a proportionate amount subtracted from these probability values will not completely deplete them.
[0406] General equations [6a]-[7a] can be used to implement the learning methodology illustrated in FIG. 13e. Given that g31(p(k))=({fraction (2/15)})p1(k) and g32(p(k))=({fraction (1/15)})p2(k), equations [6a]-[7a] can be broken down into: 20 p 3 ( k + 1 ) = p 3 ( k ) + ∑ j = 1 2 g 3 j ( p ( k ) ) = p 3 ( k ) + 2 15 p 1 ( k ) + 1 15 p 2 ( k ) ; [6a-3] p 1 ( k + 1 ) = p 1 ( k ) + g 31 ( p ( k ) ) = p 1 ( k ) - 2 15 p 1 ( k ) = 13 15 p 1 ( k ) ; and [7a-5] p 2 ( k + 1 ) = p 2 ( k ) - g 32 ( p ( k ) ) = p 2 ( k ) - 1 15 p 2 ( k ) = 14 15 p 2 ( k ) [7a-6]
[0407] FIG. 13f illustrates a methodology used to update the article probability distribution p when a triple article combination subset &agr;s3 is currently selected, and the child 605 does not succeed in matching the prompted article combination &agr; (i.e., &bgr;=1). In this case, the intuition module 715 will attempt to prevent over-challenging the child 605 by decreasing the probability that the child 605 will subsequently be prompted by the more difficult triple subset combination set &agr;s3. The intuition module 715 accomplishes this be shifting the probability distribution p from the probability value p3 to the probability values p1 and p2. Specifically, the triple article combination subset &agr;s3 is penalized by subtracting a proportionate value equal to “x” (e.g., ⅕ of p3) from probability value p3 and distributing it to the probability values p1 and p2.
[0408] Since the child's failure with a triple article combination set &agr;s1 indicates that the child 605 may not be relatively proficient at double article combinations &agr;s2, but not necessarily not proficient with the easier single article combinations &agr;s1, the probability value p2 is increased more than the probability value pi to ensure that the child 605 is not under-challenged with single article combination subsets &agr;s1. For example, the proportions of “x” added to the probability values p1 and p2 can be ⅓ and ⅔, respectively. Notably, the methodology illustrated in FIG. 13f allows control over the relative amounts that are added to the probability values p1 and p2. That is, the amount added to the probability value p2 will always be greater than the amount added to the probability value p1 irrespective of the current magnitudes of the probability values p1 and p2, thereby ensuring that the child 605 is not under-challenged with single article combination subsets &agr;s1.
[0409] General equations [20] and [21 a] can be used to implement the learning methodology illustrated in FIG. 13f. Given that h3p(k))=(⅕)p3(k), d31=⅓, and d32=⅔, equations [20] and [21a] can be broken down into: 21 p 3 ( k + 1 ) = p 3 ( k ) - h 3 ( p ( k ) ) = p 3 ( k ) - 1 5 p 3 ( k ) = 4 5 p 3 ( k ) ; [20-3] p 1 ( k + 1 ) = p 1 ( k ) + ( 1 3 ) ( 1 5 ) p 3 ( k ) = p 1 ( k ) + 1 15 p 3 ( k ) ; and [21a-5] p 2 ( k + 1 ) = p 2 ( k ) + ( 2 3 ) ( 1 5 ) p 3 ( k ) = p 2 ( k ) + 2 15 p 3 ( k ) [21a-6]
[0410] Although the intuition module 715 has been previously described as selecting the learning methodologies based merely on the difficulty of the currently prompted article combination &agr;x and the outcome value &bgr;, the intuition module 715 may base its decision on other factors, such as the current probability values px. For example, assuming a single article combination subset &agr;s1 is currently selected, and the child 605 succeeds in matching the prompted article combination &agr; (i.e., &bgr;=0), if probability value p3 is higher than probability value p2, a modified version of the learning methodology illustrated in FIG. 13a can be selected, wherein the all of the amount subtracted from probability value p1 can be added to probability value p2 to make the learning transition smoother.
[0411] Having now described the structure of the educational program 700, the steps performed by the educational program 700 will be described with reference to FIG. 14. First, the probability update module 720 initializes the article probability distribution p (step 805). For example, the educational program 700 may assume that the child 605 initially exhibits a relatively low skill level with the doll 600, in which case, the initial combined probability values px corresponding to the single article combination subset &agr;s1 can equal 0.80, and the initial combined probability values px corresponding to the double article combination subset &agr;s2 can equal 0.20. Thus, the probability distribution p is weighted towards the single article combination subset &agr;s1, so that, initially, there is a higher probability that the child 605 will be prompted with the easier single article combinations &agr;x.
[0412] The article selection module 725 then pseudo-randomly selects an article combination &agr;x from the article probability distribution px and accordingly prompts the child 605 with that selected article combination &agr;x (step 810). In the alternative case where the article probability distribution p only contains three probability values px for the respective three article combination subsets &agr;s, the article section module 725 pseudo-randomly selects an article combination subset &agr;s, and then from the selected article combination subset &agr;s, randomly selects an article combination &agr;x.
[0413] After the article combination &agr;x has been selected, the outcome evaluation module 730 then determines whether the article combination &lgr;x has been selected by the child 605, i.e., whether the child has applied the articles 610 to the doll 600 (step 815). To allow the child 605 time to apply the articles 610 to the doll 600 or to change misapplied articles 610, this determination can be made after a certain period of time has expired (e.g., 10 seconds). If an article combination &lgr;x has not been selected by the child 605 at step 815, the educational program 700 then returns to step 815 where it is again determined if an article combination &lgr;x has been selected. If an article combination &lgr;x has been selected by the child 605, the outcome evaluation module 730 then determines if it matches the article combination &agr;x prompted by the doll 600, and generates the outcome value &bgr; in response thereto (step 820).
[0414] The intuition module 715 then modifies the functionality of the probability update module 720 by selecting the learning methodology that is used to update the article probability distribution p based on the outcome value &bgr; and the number of articles contained within the prompted article combination &agr;x (step 825). Specifically, the intuition module 715 selects (1) equations [20-1], [21a-1], and [21a-2] if the article combination &lgr;x, selected by the child 605 matches a prompted single article combination &agr;x; (2) equations [6a-1], [7a-1], and [7a-2] if the article combination &lgr;x selected by the child 605 does not match a prompted single article combination &agr;x; (3) equations [20-2], [21a-3], and [21a-4] if the article combination &lgr;x selected by the child 605 matches a prompted double article combination &agr;x; (4) equations [6a-2], [7a-3], and [7a-4] if the article combination &lgr;x selected by the child 605 does not match a prompted double article combination &agr;x; (5) equations [6a-3], [7a-5], and [7a-6] if the article combination &lgr;x selected by the child 605 matches a prompted triple article combination &agr;x; and (6) equations [20-3], [21a-5], and [21a-6] if the article combination &lgr;x selected by the child 605 does not match a prompted triple article combination &agr;x.
[0415] The probability update module 720 then, using equations selected by the intuition module 715, updates the article probability distributionp(step 830). Specifically, when updating the article probability distribution p, the probability update module 720 initially treats the article probability distribution p as having three probability values p, corresponding to the three article combination subsets &agr;s. After the initial update, the probability update module 720 then evenly distributes the three updated probability values px among the probability values px corresponding to the article combinations &agr;. That is, the probability value px corresponding to the single article combination subset &agr;s1 is distributed among the probability values px corresponding to the nine single article combinations &agr;x; the probability value px corresponding to the double article combination subset &agr;s2 is distributed among the probability values px corresponding to the twenty-two double article combinations a,; and the probability value px corresponding to the triple article combination subset as3 is distributed among the probability values px corresponding to the twelve triple article combinations &agr;x. In the alternative embodiment where the article probability distribution p actually contains three article probability values px corresponding to the three article combination subsets &agr;s, the probability update module 720 simply updates the three article probability values px, which are subsequently selected by the article selection module 725. The program 700 then returns to step 810.
[0416] Although the actions on which the program 700 operates has previously been described as related to prompted tasks, e.g., article combinations, the actions can also relate to educational games that can be played by the child 605. Another single-player educational program 900 (shown in FIG. 15) developed in accordance with the present inventions is described in the context of a modification of the previously described child's learning toy 600 (shown in FIG. 11).
[0417] The modified doll 600 can contain three educational games (represented by games &agr;1-&agr;3) that can be presented to the child 605. These educational games will have different degrees of difficulty. For example, the first game &agr;1 can be a relatively easy article matching game that prompts the child 605 to apply the articles one at a time to the doll 600. The second game &agr;2 can be a more difficult color matching memory game that prompts the child 605 with a series of colors that the child 605 could input using a color keypad (not shown). The third game &agr;3 can be an even more difficult cognition game that prompts the child 605 with a number that the child 605 responds with color coded numbers the sum of which should add up to the prompted number.
[0418] In this case, the doll 600 seeks to challenge the child 605 by presenting him or her with more difficult games as the child 605 masters the doll 600. For example, if the child 605 exhibits a proficiency at the article matching game &agr;1, the doll 600 will less frequently present the child 605 with the article matching game &agr;1, and more frequently present the child 605 with color matching memory game &agr;2 and cognition game &agr;3. If the child 605 exhibits a proficiency at the color matching memory game &agr;2, the doll 600 will less frequently present the child 605 with the article matching game al and color matching memory game &agr;2, and more frequently present the child 605 with the cognition game &agr;3. If the child 605 exhibits a proficiency at the cognition game &agr;3, the doll 600 will even more frequently present the cognition game &agr;3 to the child 605.
[0419] The doll 600 also seeks to avoid over challenging the child 605 and frustrating the learning process. For example, if the child 605 does not exhibit a proficiency at the cognition game &agr;3, the doll 600 will less frequently present the child 605 with the cognition game &agr;3 and more frequently present the child 605 with the article matching game &agr;1 and color matching memory game &agr;2. If the child 605 does not exhibit a proficiency at the color matching memory game &agr;2, the doll 600 will less frequently present the child 605 with the color matching memory game &agr;2 and cognition game &agr;3, and more frequently present the child 605 with the article matching game &agr;1. If the child 605 does not exhibit a proficiency at the article matching game &agr;1, the doll 600 will even more frequently present the article matching game &agr;1 to the child 605.
[0420] The educational program 900 is similar to the previously described educational program 700 with the exception that it treats the actions &agr;x as educational games, rather than article combinations, and treats the child actions &lgr;x as actions to be input by the child 605 as specified by the currently played educational game, i.e., inputting articles in the case of the article matching game &agr;1, inputting colors in the case of the color matching memory game &agr;2, and inputting number coded colors in the case of the cognition game &agr;3.
[0421] To this end, the educational program 900 generally includes a probabilistic learning module 910 and an intuition module 915, which are specifically tailored for the modified doll 600. The probabilistic learning module 910 comprises a probability update module 920, a game selection module 925, and an outcome evaluation module 930. Specifically, the probability update module 920 is mainly responsible for learning the child's current skill level, with the outcome evaluation module 930 being responsible for evaluating the educational games &agr;x presented by the doll 600 relative to the actions &lgr;x selected by the child 605. The game selection module 925 is mainly responsible for using the learned skill level of the child 605 to select the games &agr;x that presented to the child 605. The intuition module 915 is responsible for directing the learning of the educational program 900 towards the objective, and specifically, dynamically pushing the skill level of the child 605 to a higher level. In this case, the intuition module 915 operates on the probability update module 920, and specifically selects the methodology that the probability update module 920 will use to update a game probability distribution p.
[0422] To this end, the outcome evaluation module 930 is configured to receive an educational game &agr;x from the game selection module 925 (i.e., one of the three educational games to be presented to the child 605 by the doll 600), and receive actions &lgr;x from the child 605 (i.e., actions that the child 605 inputs into doll 600 during the current educational game &agr;x). The outcome evaluation module 930 is also configured to determine whether the actions &lgr;x received from the child 605 are successful within the selected educational game &agr;x, with the outcome value &bgr; equaling one of two predetermined values, e.g., “0” if the child actions &lgr;x are successful within the selected educational game &agr;x, and “1” if the child actions &lgr;x is not successful within the selected educational game &agr;x. In this case, a P-type learning methodology is used. Optionally, Q- and S-type learning methodologies can be used to quantify child actions &lgr;x that are relatively successful or unsuccessful.
[0423] The probability update module 920 is configured to generate and update the game probability distribution p in a manner directed by the intuition module 915, with the article probability distribution p containing three probability values px corresponding to the three educational games &agr;x. The game selection module 925 is configured for receiving the article probability distribution p from the probability update module 920, and pseudo-randomly selecting the education game &agr;x therefrom in the same manner as the article selection module 725 of the program 700 selects article combination subsets &agr;s.
[0424] The intuition module 915 is configured to modify the functionality of the probability update module 920 based on the performance index &phgr;, and in this case, the current skill level of the child 605 relative to the current teaching level of the doll 600. In the preferred embodiment, the performance index &phgr; is quantified in terms of the degree of difficulty of the currently selected educational game &agr;x and the outcome value &bgr; (i.e., whether or not the actions &lgr;x selected by the child 605 are successful). In this respect, the performance index &phgr; is instantaneous. It should be appreciated, however, that the performance of the educational program 900 can also be based on a cumulative performance index &phgr;. For example, the educational program 900 can keep track of a percentage of the child's successful with the educational games &agr;x.
[0425] The intuition module 915 modifies the functionality of the probability update module 920 is the same manner as the previously described intuition module 715 modifies the functionality of the probability update module 720. That is, the intuition module 915 determines which updating methodology will be used and which educational game a will be rewarded or penalized in a manner similar to that described with respect to FIGS. 13a-f. For example, the intuition module 915 directs the probability update module 920 to shift the game probability distribution p from probability value(s) px corresponding to educational games &agr;x associated with lesser difficult levels to probability value(s) px corresponding to educational games &agr;x associated with greater difficult levels when the child 605 is relatively successful at the currently selected education game &agr;x, and to shift the game probability distribution p from probability value(s) p, corresponding to educational games &agr;x associated with greater difficult levels to probability value(s) px corresponding to educational games &agr;x associated with lesser difficult levels when the child 605 is relatively unsuccessful at the currently selected education game &agr;x.
[0426] In the illustrated embodiment, P-type learning methodologies (&bgr; equals either “0” or “1”) are used, in which case, it is assumed that the child 605 is absolutely successful or unsuccessful in any given educational game &agr;x. Alternatively, Q- and S-type learning methodologies (&bgr; is between “0” and “1”) are used, in which case, it is assumed that the child 605 can be partially successful or unsuccessful in any given educational game &agr;x. For example, the outcome value &bgr; may be a lesser value if most of the child actions &lgr;x are successful, and may be a greater value if most of the child actions &lgr;x are unsuccessful.
[0427] The intuition module 915 can select from the learning methodologies illustrated in FIGS. 13a-f. For example, the intuition module 915 can select (1) the methodology illustrated in FIG. 13a if the child 605 succeeds in the article matching game &agr;1; (2) the methodology illustrated in FIG. 13b if the child 605 does not succeed in the article matching game &agr;1; (3) the methodology illustrated in FIG. 13c if the child 605 succeeds in the color matching memory game &agr;2; (4) the methodology illustrated in FIG. 13d if the child 605 does not succeed in the color matching memory game &agr;2; (5) the methodology illustrated in FIG. 13e if the child 605 succeeds in the cognition game &agr;3; and (6) the methodology illustrated in FIG. 13f if the child 605 does not succeed in the cognition game &agr;3.
[0428] So that selection of the educational games &agr;x is not too erratic, the intuition module 915 may optionally modify the game selection module 925, so that it does not select the relatively easy article matching game &agr;1 after the relatively difficult cognition game &agr;3 has been selected, and does not select the relatively difficult cognition game &agr;3 after the relatively easy article matching game &agr;x has been selected. Thus, the teaching level of the doll 600 will tend to play the article matching game &agr;1, then the color matching memory game &agr;2, and then the cognition game &agr;3, as the child 605 learns.
[0429] Having now described the structure of the educational program 900, the steps performed by the educational program 900 will be described with reference to FIG. 16. First, the probability update module 920 initializes the game probability distribution p (step 1005). For example, the educational program 900 may assume that the child 605 initially exhibits a relatively low skill level with the doll 600, in which case, the initial probability values px corresponding to the relatively easy article matching game &agr;x can equal 0.80, and the initial probability value p, corresponding to the color matching memory game &agr;2 can equal 0.20. Thus, the probability distribution p is weighted towards the article matching game &agr;1, so that, initially, there is a higher probability that the child 605 will be prompted with the easier article matching game &agr;x.
[0430] The game selection module 925 then pseudo-randomly selects an educational game &agr;x from the game probability distribution px and accordingly presents the child 605 with that selected game &agr;x (step 1010).
[0431] After the educational game &agr;x has been selected, the outcome evaluation module 930 then receives actions &lgr;x from the child 605 (step 1015) and determines whether the game &agr;x has been completed (step 1020). If the selected educational game &agr;x has not been completed at step 1015, the educational program 900 then returns to step 1015 where it receives actions &lgr;x from the child 605. If the selected educational game &agr;x has been completed at step 1015, the outcome evaluation module 930 then determines whether the actions &lgr;x from the child 605 are successful, and generates the outcome value &bgr; in response thereto (step 1025).
[0432] The intuition module 915 then modifies the functionality of the probability update module 920 by selecting the learning methodology that is used to update the article probability distribution p based on the outcome value &bgr; and the currently played educational game &agr;x (step 1030). Specifically, the intuition module 915 selects (1) equations [20-1], [21a-1], and [21a-2] if the actions &lgr;x selected by the child 605 within the article matching game &agr;x are relatively successful; (2) equations [6a-1], [7a-1], and [7a-2] if the actions &lgr;x selected by the child 605 within the article matching game &agr;x are relatively unsuccessful; (3) equations [20-2], [21a-3], and [21a-4] if the actions &lgr;x selected by the child 605 within the color matching memory game &agr;2 are relatively successful; (4) equations [6a-2], [7a-3], and [7a-4] if the actions &lgr;x selected by the child 605 within the color matching memory game &agr;2 are relatively unsuccessful; (5) equations [6a-3], [7a-5], and [7a-6] if the actions &lgr;x selected by the child 605 within the cognition game &agr;3 are relatively successful; and (6) equations [20-3], [21a-5], and [21a-6] if the actions Ax selected by the child 605 within the cognition game &agr;3 are relatively unsuccessful.
[0433] The probability update module 920 then, using equations selected by the intuition module 915, updates the article probability distribution p (step 1035). The program 900 then returns to step 1010 where the game selection module 925 again pseudo-randomly selects an educational game &agr;x from the game probability distribution px and accordingly presents the child 605 with that selected game &agr;x.
[0434] More Specific details on the above-described operation of the toy 600 can be found in the computer program listing appendix attached hereto and previously incorporated herein by reference. It is noted that the file “Intuition Intelligence-simonsays.doc” shows that the game can be played in two modes.
[0435] Specifically, in the hardware mode (communication through USB), the toy is connected to a digital logic board on a USB Controller board. A USB cable connects the USB port on the PC and the USB Controller board. A simple USB software driver on the PC aids in reading the “code” that is generated by the digital logic. The digital logic is connected to the various switches and the sensor points of the toy. The sensors are open circuits that are closed when an accessory is placed (or connected to a sensor) on the toy. Each article or accessory of the toy has different resistor values. The digital logic determines which sensors circuits are closed and open, which switches are ON and OFF, and the resistor value of the article connected to the sensor. Based on these inputs digital logic generates different codes. Digital logic generated code is processed by the program in the PC.
[0436] In the software mode, the hardware communication is simulated by typing in the code directly to a text box. The software version emulation eliminates the need for USB communication and the digital logic circuit code generation. The code that is needed for the game play is pre-initialized in variables for different prompts. The code that is expected by the toy is also shown on the screen, so that the toy can be tested. If the code expected and the code typed in the text box (or the hardware generated code) are the same, it is consider a success for the child.
[0437] Single-User Phone Number Listing Program
[0438] Although game and toy applications have only been described in detail so far, the learning program 100 can have even more applications. For example, referring to FIGS. 17 and 18, a priority listing program 1200 (shown in FIG. 19) developed in accordance with the present inventions is described in the context of a mobile phone 1100. The mobile phone 1100 comprises a display 1110 for displaying various items to a phone user 1115 (shown in FIG. 19). The mobile phone 1100 further comprises a keypad 1140 through which the phone user 1115 can dial phone numbers and program the functions of the mobile phone 1100. To the end, the keypad 1140 includes number keys 1145, a scroll key 1146, and selection keys 1147. The mobile phone 1100 further includes a speaker 1150, microphone 1155, and antenna 1160 through which the phone user 1115 can wirelessly carry on a conversation. The mobile phone 1100 further includes keypad circuitry 1170, control circuitry 1135, memory 1130, and a transceiver 1165.
[0439] The keypad circuitry 1170 decodes the signals from the keypad 1140, as entered by the phone user 1115, and supplies them to the control circuitry 1135. The control circuitry 1135 controls the transmission and reception of call and voice signals. During a transmission mode, the control circuitry 1135 provides a voice signal from the microphone 1155 to the transceiver 1165. The transceiver 1165 transmits the voice signal to a remote station (not shown) for communication through the antenna 1160. During a receiving mode, the transceiver 1165 receives a voice signal from the remote station through the antenna 1160. The control circuitry 1135 then provides the received voice signal from the transceiver 1165 to the speaker 1150, which provides audible signals for the phone user 1115. The memory 1130 stores programs that are executed by the control circuitry 1135 for basic functioning of the mobile phone 1100. In many respects, these elements are standard in the industry, and therefore their general structure and operation will not be discussed in detail for purposes of brevity.
[0440] In addition to the standard features that typical mobile phones have, however, the mobile phone 1100 displays a favorite phone number list 1120 from which the phone user 1115 can select a phone number using the scroll and select buttons 1146 and 1147 on the keypad 1140. In the illustrated embodiment, the favorite phone number list 1120 has six phone numbers 1820 at any given time, which can be displayed to the phone user 1115 in respective sets of two and four numbers. It should be noted, however, that the total number of phone numbers within the list 1120 may vary and can be displayed to the phone user 1115 in any variety of manners.
[0441] The priority listing program 1200, which is stored in the memory 1130 and executed by the control circuitry 1135, dynamically updates the telephone number list 1120 based on the phone user's 1115 current calling habits. For example, the program 1200 maintains the favorite phone number list 1120 based on the number of times a phone number has been called, the recent activity of the called phone number, and the time period (e.g., day, evening, weekend, weekday) in which the phone number has been called, such that the favorite telephone number list 1120 will likely contain a phone number that the phone user 1115 is anticipated to call at any given time. As will be described in further detail below, the listing program 1200 uses the existence or non-existence of a currently called phone number on a comprehensive phone number list as a performance index &phgr; in measuring its performance in relation to its objective of ensuring that the favorite phone number list 1120 will include future called phone numbers, so that the phone user 1115 is not required to dial the phone number using the number keys 1145. In this regard, it can be said that the performance index &phgr; is instantaneous. Alternatively or optionally, the listing program 1200 can also use the location of the phone number on the comprehensive phone number list as a performance index &phgr;.
[0442] Referring now to FIG. 19, the listing program 1200 generally includes a probabilistic learning module 1210 and an intuition module 1215, which are specifically tailored for the mobile phone 1100. The probabilistic learning module 1210 comprises a probability update module 1220, a phone number selection module 1225, and an outcome evaluation module 1230. Specifically, the probability update module 1220 is mainly responsible for learning the phone user's 1115 calling habits and updating a comprehensive phone number list a that places phone numbers in the order that they are likely to be called in the future during any given time period. The outcome evaluation module 1230 is responsible for evaluating the comprehensive phone number list a relative to current phone numbers &lgr;x called by the phone user 1115. The phone number selection module 1225 is mainly responsible for selecting a phone number subset &agr;x from the comprehensive phone number list &agr; for eventual display to the phone user 1115 as the favorite phone number list 1120. The intuition module 1215 is responsible for directing the learning of the listing program 1200 towards the objective, and specifically, displaying the favorite phone number list 1120 that is likely to include the phone user's 1115 next called phone number. In this case, the intuition module 1215 operates on the probability update module 1220, the details of which will be described in further detail below.
[0443] To this end, the phone number selection module 1225 is configured to receive a phone number probability distribution p from the probability update module 1220, which is similar to equation [1] and can be represented by the following equation:
p(k)=[p1(k),p2(k),p3(k) . . . pn(k)], [1-2]
[0444] where px is the probability value assigned to a specific phone number &agr;x; n is the number of phone numbers &agr;x within the comprehensive phone number list &agr;x and k is the incremental time at which the phone number probability distribution p was updated. Based on the phone number probability distribution p, the phone number selection module 1225 generates the comprehensive phone number list &agr;, which contains the listed phone numbers &agr;x ordered in accordance with their associated probability values px. For example, the first listed phone number &agr;x will be associated with the highest probability value px, while the last listed phone number &agr;x will be associated with the lowest probability value px. Thus, the comprehensive phone number list a contains all phone numbers ever called by the phone user 1115 and is unlimited. Optionally, the comprehensive phone number list &agr; can contain a limited amount of phone numbers, e.g., 100, so that the memory 1130 is not overwhelmed by seldom called phone numbers. In this case, seldom called phone numbers &agr;x may eventually drop off of the comprehensive phone number list &agr;.
[0445] It should be noted that a comprehensive phone number list &agr; need not be separate from the phone number probability distribution p, but rather the phone number probability distribution p can be used as the comprehensive phone number list &agr; to the extent that it contains a comprehensive list of phone numbers &agr;x corresponding to all of the called phone numbers &lgr;x. However, it is conceptually easier to explain the aspects of the listing program 1200 in the context of a comprehensive phone number list that is ordered in accordance with the corresponding probability values px, rather than in accordance with the order in which they are listed in the phone number probability distribution p.
[0446] From the comprehensive phone number list &agr;, the phone number selection module 1225 selects the phone number subset &agr;s (in the illustrated embodiment, six phone numbers &agr;x) that will be displayed to the phone user 1115 as the favorite phone number list 1120. In the preferred embodiment, the selected phone number subset &agr;s will contain those phone numbers &agr;x that correspond to the highest probability values px, i.e., the top six phone numbers &agr;x on the comprehensive phone number list &agr;.
[0447] As an example, consider Table 7, which sets forth in exemplary comprehensive phone number list &agr; with associated probability values px. 7 TABLE 7 Exemplary Probability Values for Comprehensive Phone Number List Number Listed Phone Numbers (&agr;1) Probability Values (p1) 1 949-339-2932 0.253 2 343-3985 0.183 3 239-3208 0.128 4 239-2908 0.102 5 343-1098 0.109 6 349-0085 0.073 7 239-3833 0.053 8 239-4043 0.038 . . . . . . . . . 96 213-483-3343 0.009 97 383-303-3838 0.007 98 808-483-3984 0.007 99 398-3838 0.005 100 239-3409 0.002
[0448] In this exemplary case, phone numbers 949-339-2932, 343-3985, 239-3208, 239-2908, 343-1098, and 349-0085 will be selected as the favorite phone number list 1220, since they are associated with the top six probability values px.
[0449] The outcome evaluation module 1230 is configured to receive a called phone number &lgr;x from the phone user 1115 via the keypad 1140 and the comprehensive phone number list &agr; from the phone number selection module 1225. For example, the phone user 1115 can dial the phone number &lgr;x using the number keys 1145 of the keypad 1140, selecting the phone number &lgr;x, from the favorite phone number list 1120 by operating the scroll and selection keys 1146 and 1147 of the keypad 1140, or through any other means. In this embodiment, the phone number &lgr;x, can be selected from a virtually infinite set of phone numbers &lgr;, i.e., all valid phone numbers that can be called by the mobile phone 1100. The outcome evaluation module 1230 is further configured to determine and output an outcome value &bgr; that indicates if the currently called phone number &lgr;x is on the comprehensive phone number list a. In the illustrated embodiment, the outcome value &bgr; equals one of two predetermined values: “1” if the currently called phone number &lgr;x matches a phone number &agr;x on the comprehensive phone number list &agr;x and “0” if the currently called phone number &lgr;x does not match a phone number &agr;x on the comprehensive phone number list a.
[0450] It can be appreciated that unlike in the duck game 300 where the outcome value &bgr; is partially based on the selected game move &agr;i, the outcome value &bgr; is technically not based on listed phone numbers &agr;x selected by the phone number selection module 1225, i.e., the phone number subset &agr;s, but rather whether a called phone number &lgr;x is on the comprehensive phone number list &agr; irrespective of whether it is in the phone number subset &agr;s. It should be noted, however, that the outcome value &bgr; can optionally or alternatively be partially based on the selected phone number subset &agr;s, as will be described in further detail below.
[0451] The intuition module 1215 is configured to receive the outcome value &bgr; from the outcome evaluation module 1230 and modify the probability update module 1220, and specifically, the phone number probability distribution p, based thereon. Specifically, if the outcome value &bgr; equals “0,” indicating that the currently called phone number &lgr;x was not found on the comprehensive phone number list &agr;, the intuition module 1215 adds the called phone number &lgr;x to the comprehensive phone number list &agr; as a listed phone number &agr;x.
[0452] The phone number &agr;x can be added to the comprehensive phone number list a in a variety of ways. In general, the location of the added phone number &agr;x within the comprehensive phone number list a depends on the probability value px assigned or some function of the probability value px assigned.
[0453] For example, in the case where the number of phone numbers &agr;x is not limited or has not reached its limit, the phone number &agr;x may be added by assigning a probability value px to it and renormalizing the phone number probability distribution p in accordance with the following equations: 22 p i ( k + 1 ) = f ( x ) ; [ 22 ] p j ( k + 1 ) = p j ( k ) ( 1 - f ( x ) ) ; j ≠ i [ 23 ]
[0454] where i is the added index corresponding to the newly added phone number &agr;x, px is the probability value corresponding to phone number &agr;x added to the comprehensive phone number list &agr;, f(x) is the probability value px assigned to the newly added phone number &agr;x, pj is each probability value corresponding to the remaining phone numbers &agr;j on the comprehensive phone number list &agr;, and k is the incremental time at which the phone number probability distribution was updated.
[0455] In the illustrated embodiment, the probability value px assigned to the added phone number &agr;x is simply the inverse of the number of phone numbers &agr;x on the comprehensive phone number list &agr;x and thus f(x) equals 1/(n+1), where n is the number of phone numbers on the comprehensive phone number list a prior to adding the phone number &agr;x. Thus, equations [22] and [23] break down to: 23 p i ( k + 1 ) = 1 n + 1 ; [22-1] p j ( k + 1 ) = p j ( k ) n n + 1 ; j ≠ i [23-1]
[0456] In the case where the number of phone numbers &agr;x is limited and has reached its limit, the phone number &agr; with the lowest corresponding priority value px is replaced with the newly called phone number &lgr;x by assigning a probability value px to it and renormalizing the phone number probability distribution p in accordance with the following equations:
px(k+1)=f(x) [24]; 24 p j ( k + 1 ) = p j ( k ) ∑ j ≠ i n p j ( k ) ( 1 - f ( x ) ) ; j ≠ i [25]
[0457] where i is the index used by the removed phone number &agr;x, px is the probability value corresponding to phone number &agr;x added to the comprehensive phone number list &agr;,f(x) is the probability value pm assigned to the newly added phone number &agr;x, pj is each probability value corresponding to the remaining phone numbers &agr;j on the comprehensive phone number list &agr;x and k is the incremental time at which the phone number probability distribution was updated.
[0458] As previously stated, in the illustrated embodiment, the probability value px assigned to the added phone number &agr;x is simply the inverse of the number of phone numbers &agr;x on the comprehensive phone number list &agr;x and thus f(x) equals 1/n, where n is the number of phone numbers on the comprehensive phone number list &agr;. Thus, equations [24] and [25] break down to: 25 p i ( k + 1 ) = 1 n ; [24-1] p j ( k + 1 ) = p j ( k ) ∑ j ≠ i n p j ( k ) ( n - 1 n ) ; j ≠ i [25-1]
[0459] It should be appreciated that the speed in which the automaton learns can be controlled by adding the phone number &agr;x to specific locations within the phone number probability distribution p. For example, the probability value px assigned to the added phone number &agr;x can be calculated as the mean of the current probability values px, such that the phone number &agr;x will be added to the middle of the comprehensive phone number list &agr; to effect an average learning speed. The probability value px assigned to the added phone number &agr;x can be calculated as an upper percentile (e.g. 25%) to effect a relatively quick learning speed. Or the probability value px assigned to the added phone number &agr;x can be calculated as a lower percentile (e.g. 75%) to effect a relatively slow learning speed. It should be noted that if there is a limited number of phone numbers &agr;x on the comprehensive phone number list &agr;x thereby placing the lowest phone numbers &agr;x in the likelihood position of being deleted from the comprehensive phone number list &agr;, the assigned probability value px should be not be so low as to cause the added phone number &agr;i to oscillate on and off of the comprehensive phone number list &agr; when it is alternately called and not called.
[0460] In any event, if the outcome value &bgr; received from the outcome evaluation module 1230 equals “1,” indicating that the currently called phone number &lgr;x was found on the comprehensive phone number list &agr;, the intuition module 1215 directs the probability update module 1220 to update the phone number probability distribution p using a learning methodology. In the illustrated embodiment, the probability update module 1220 utilizes a linear reward-inaction P-type update.
[0461] As an example, assume that a currently called phone number &lgr;x, matches a phone number &agr;10 on the comprehensive phone number list &agr;, thus creating an outcome value &bgr;=1. Assume also that the comprehensive phone number list a currently contains 50 phone numbers &agr;x. In this case, general updating equations [6] and [7] can be expanded using equations [10] and [11], as follows: 26 p 10 ( k + 1 ) = p 10 ( k ) + ∑ j = 1 j ≠ 10 50 ap j ( k ) ;
p1(k+1)=p1(k)−ap1(k);
p2(k+1)=p2(k)−ap2(k);,
p4(k+1)=p4(k)−ap4(k);
p50(k+1)=p50(k)−ap50(k)
[0462] Thus, the corresponding probability value p10 is increased, and the phone number probability values px corresponding to the remaining phone numbers &agr;x are decreased. The value of a is selected based on the desired learning speed. The lower the value of &agr;x the slower the learning speed, and the higher the value of &agr;x the higher the learning speed. In the preferred embodiment, the value of a has been chosen to be 0.02. It should be noted that the penalty updating equations [8] and [9] will not be used, since in this case, a reward-penalty P-type update is not used.
[0463] Thus, it can be appreciated that, in general, the more a specific listed phone number &agr;x is called relative to other listed phone numbers &agr;x, the more the corresponding probability value p, is increased, and thus the higher that listed phone number &agr;x is moved up on the comprehensive phone number list &agr;. As such, the chances that the listed phone number &agr;x will be contained in the selected phone number subset &agr;s and displayed to the phone user 1115 as the favorite phone number list 1120 will be increased. In contrast, the less a specific listed phone number &agr;x is called relative to other listed phone numbers &agr;x, the more the corresponding probability value p, is decreased (by virtue of the increased probability values px corresponding to the more frequently called listed phone numbers &agr;x), and thus the lower that listed phone number &agr;x is moved down on the comprehensive phone number list &agr;. As such, the chances that the listed phone number &agr;x will be contained in the phone number subset &agr;s selected by the phone number selection module 1225 and displayed to the phone user 1115 as the favorite phone number list 1120 will be decreased.
[0464] It can also be appreciated that due to the nature of the learning automaton, the relative movement of a particular listed phone number &agr;x is not a matter of how many times the phone number &agr;x is called, and thus, the fact that the total number of times that a particular listed phone number &agr;x has been called is high does not ensure that it will be contained in the favorite phone number list 1120. In reality, the relative placement of a particular listed phone number &agr;x within the comprehensive phone number list &agr;s is more of a function of the number of times that the listed phone number &agr;x has been recently called. For example, if the total number of times a listed phone number &agr;x is called is high, but has not been called in the recent past, the listed phone number &agr;x may be relatively low on the comprehensive phone number list &agr; and thus it may not be contained in the favorite phone number list 1120. In contrast, if the total number of times a listed phone number &agr;x is called is low, but it has been called in the recent past, the listed phone number &agr;x may be relatively high on the comprehensive phone number list &agr; and thus it may be contained in the favorite phone number list 1120. As such, it can be appreciated that the learning automaton quickly adapts to the changing calling patterns of a particular phone user 1115.
[0465] It should be noted, however, that a phone number probability distribution p can alternatively be purely based on the frequency of each of the phone numbers &lgr;x. For example, given a total of n phone calls made, and a total number of times that each phone number is received f1, f2, f3 . . . , the probability values px for the corresponding listed phone calls &agr;i can be: 27 p i ( k + 1 ) = fi n [ 26 ]
[0466] Noteworthy, each probability value px is not a function of the previous probability value px (as characterized by learning automaton methodology), but rather the frequency of the listed phone number &agr;x and total number of phone calls n. With the purely frequency-based learning methodology, when a new phone number &agr;x is added to the phone list &agr;x its corresponding probability value px will simply be 1/n, or alternatively, some other function of the total number of phone calls n. Optionally, the total number of phone calls n is not absolute, but rather represents the total number of phone calls n made in a specific time period, e.g., the last three months, last month, or last week. In other words, the phone number probability distribution p can be based on a moving average. This provides the frequency-based learning methodology with more dynamic characteristics.
[0467] In any event, as described above, a single comprehensive phone number list &agr; that contains all phone numbers called regardless of the time and day of the week is generated and updated. Optionally, several comprehensive phone number lists a can be generated and updated based on the time and day of the week. For example, Tables 8 and 9 below set forth exemplary comprehensive phone number lists &agr;1 and &agr;2 that respectively contain phone numbers a1, and &agr;2i are called during the weekdays and weekend. 8 TABLE 8 Exemplary Probability Values for Comprehensive Weekday Phone Number List Listed Weekday Phone Number Numbers (&agr;11) Probability Values (p1) 1 349-0292 0.223 2 349-0085 0.213 3 343-3985 0.168 4 343-2922 0.122 5 328-2302 0.111 6 928-3882 0.086 7 343-1098 0.073 8 328-4893 0.032 . . . . . . . . . 96 493-3832 0.011 97 383-303-3838 0.005 98 389-3898 0.005 99 272-3483 0.003 100 213-483-3343 0.001
[0468] 9 TABLE 9 Exemplary Probability Values for Comprehensive Weekend Phone Number List Listed Weekend Phone Number Numbers (&agr;21) Probability Values (p1) 1 343-3985 0.238 2 343-1098 0.194 3 949-482-2382 0.128 4 343-2922 0.103 5 483-4838 0.085 6 349-0292 0.073 7 349-4929 0.062 8 493-4893 0.047 . . . . . . . . . 96 202-3492 0.014 97 213-403-9232 0.006 98 389-3893 0.003 99 272-3483 0.002 100 389-3898 0.001
[0469] Notably, the top six locations of the exemplary comprehensive phone number lists &agr;1 and &agr;2 contain different phone numbers &agr;1x and &agr;2x, presumably because certain phone numbers &agr;1x(e.g., 349-0085, 328-2302, and 928-3882) were mostly only called during the weekdays, and certain phone numbers &agr;2x (e.g., 343-1098, 949-482-2382 and 483-4838) were mostly only called during the weekends. The top six locations of the exemplary comprehensive phone number lists &agr;1 and &agr;2 also contain common phone numbers &agr;1x and &agr;2x, presumably because certain phone numbers &agr;1x and &agr;2x (e.g., 349-0292, 343-3985, and 343-2922) were called during the weekdays and weekends. Notably, these common phone numbers &agr;1x and &agr;2x are differently ordered in the exemplary comprehensive phone number lists &agr;1 and &agr;2, presumably because the phone user's 1115 weekday and weekend calling patterns have differently influenced the ordering of these phone numbers. Although not shown, the comprehensive phone number lists &agr;1 and &agr;2 can be further subdivided, e.g., by day and evening.
[0470] When there are multiple comprehensive phone number lists a that are divided by day and/or time, the phone selection module 1225, outcome evaluation module 1230, probability update module 1220, and intuition module 1215 operate on the comprehensive phone number lists &agr; based on the current day and/or time (as obtained by a clock or calendar stored and maintained by the control circuitry 1135). Specifically, the intuition module 1215 selects the particular comprehensive list &agr; that will be operated on. For example, during a weekday, the intuition module 1215 will select the comprehensive phone number lists &agr;1, and during the weekend, the intuition module 1215 will select the comprehensive phone number lists &agr;2.
[0471] The phone selection module 1225 will maintain the ordering of all of the comprehensive phone number lists &agr;, but will select the phone number subset &agr;s from the particular comprehensive phone number lists &agr; selected by the intuition module 1215. For example, during a weekday, the phone selection module 1225 will select the favorite phone number list &agr;s from the comprehensive phone number list &agr;1, and during the weekend, the phone selection module 1225 will select the favorite phone number list &agr;x from the comprehensive phone number list &agr;2. Thus, it can be appreciated that the particular favorite phone number list 1120 displayed to the phone user 1115 will be customized to the current day, thereby increasing the chances that the next phone number &lgr;x called by the phone user 1115 will be on the favorite phone number list 1120 for convenient selection by the phone user 1115.
[0472] The outcome evaluation module 1230 will determine if the currently called phone number &lgr;x matches a phone number &agr;x contained on the comprehensive phone number list a selected by the intuition module 1215 and generate an outcome value &bgr; based thereon, and the intuition module 1215 will accordingly modify the phone number probability distribution p corresponding to the selected comprehensive phone number list &agr;. For example, during a weekday, the outcome evaluation module 1230 determines if the currently called phone number &lgr;x matches a phone number &agr;x contained on the comprehensive phone number list &agr;1, and the intuition module 1215 will then modify the phone number probability distribution p corresponding to the comprehensive phone number list &agr;1. During a weekend, the outcome evaluation module 1230 determines if the currently called phone number &lgr;x matches a phone number &agr;x contained on the comprehensive phone number list &agr;2, and the intuition module 1215 will then modify the phone number probability distribution p corresponding to the comprehensive phone number list &agr;2.
[0473] In the illustrated embodiment, the outcome evaluation module 1230, probability update module 1220, and intuition module 1215 only operate on the comprehensive phone number list &agr; and were not concerned with the favorite phone number list &agr;s. It was merely assumed that a phone number &agr;x corresponding to a frequently and recently called phone number &agr;x that was not currently in the selected phone number subset &agr;s would eventually work its way into the favorite phone number list 1120, and a phone number &agr;x corresponding to a seldom called phone number a, that was currently in the selected phone number subset &agr;x would eventually work its way off of the favorite phone number list 1120.
[0474] Optionally, the outcome evaluation module 1230, probability update module 1220, and intuition module 1215 can be configured to provide further control over this process to increase the chances that the next called phone number &lgr;x will match a phone number &agr;x in the selected phone number list &agr;s for display to the user 1115 as the favorite phone number list 1120.
[0475] For example, the outcome evaluation module 1230 may generate an outcome value &bgr; equal to “1” if the currently called phone number &lgr;x matches a phone number &agr;x in the previously selected phone number subset &agr;s, “0” if the currently called phone number &lgr;x does not match a phone number &agr;x on the comprehensive phone number list &agr;, and “2” if the currently called phone number &lgr;x matches a phone number &agr;x on the comprehensive phone number list &agr;, but not in the previously selected phone number subset &agr;s. If the outcome value is “0” or “1”, the intuition module 1215 will direct the probability update module 1220 as previously described. If the outcome value is “2”, however, the intuition module 1215 will not direct the probability update module 1220 to update the phone number probability distribution p using a learning methodology, but instead will assign a probability value px to the listed phone number &agr;x. For example, the assigned probability value px may be higher than that corresponding to the last phone number &agr;x in the selected phone number subset &agr;s, in effect, replacing that last phone number &agr;x with the listed phone number &agr;x corresponding to the currently called phone number &lgr;x. The outcome evaluation module 1230 may generate an outcome value equal to other values, e.g., “3” if the a phone number &lgr;x corresponding to a phone number &agr;x not in the selected phone number subset &agr;s has been called a certain number of times within a defined period, e.g., 3 times in one day or 24 hours. In this case, the intuition module 1215 may direct the probability update module 1220 to assign a probability value px to the listed phone number &agr;x, perhaps placing the corresponding phone number &agr;x on the favorite phone number list &agr;s.
[0476] As another example to provide better control over the learning process, the phone number probability distribution p can be subdivided into two sub-distributions p1 and P2, with the first sub-distribution p1 corresponding to the selected phone number subset &agr;s, and the second sub-distribution p2 corresponding to the remaining phone numbers &agr;x on the comprehensive phone number list &agr;. In this manner, the first and second sub-distributions p1 and p2 will not affect each other, thereby preventing the relatively high probability values px corresponding to the favorite phone number list &agr;s from overwhelming the remaining probability values px, which might otherwise slow the learning of the automaton. Thus, each of the first and second sub-distributions p1 and p2 are independently updated with the same or even different learning methodologies. Modification of the probability update module 1220 can be accomplished by the intuition module 1215 in the foregoing manners.
[0477] The intuition module 1215 may also prevent any one probability value px from overwhelming the remaining probability values px by limiting it to a particular value, e.g., 0.5. In this sense, the learning module 1210 will not converge to any particular probability value px, which is not the objective of the mobile phone 1100. That is, the objective is not to find a single favorite phone number, but rather a list of favorite phone numbers that dynamically changes with the phone user's 1115 changing calling patterns. Convergence to a single probability value px would defeat this objective.
[0478] So far, it has been explained that the listing program 1200 uses the instantaneous outcome value &bgr; as a performance index &phgr; in measuring its performance in relation to its objective of maintaining favorite phone number list 1120 to contain future called telephone numbers. It should be appreciated, however, that the performance of the listing program 1200 can also be based on a cumulative performance index &phgr;. For example, the listing program 1200 can keep track of a percentage of the called phone numbers &agr;x that match phone numbers &agr;x in the selected phone number subset &agr;s or a consecutive number of called phone numbers &lgr;x that do not match phone numbers &agr;x not found in the selected phone number subset &agr;s, based on the outcome value &bgr;, e.g., whether the outcome value &bgr; equals “2.” Based on this cumulative performance index &phgr;, the intuition module 1215 can modify the learning speed or nature of the learning module 1210.
[0479] It has also been described that the phone user 1115 actions encompass phone numbers &lgr;x from phone calls made by the mobile phone 1100 (i.e., outgoing phone calls) that are used to generate the outcome values &bgr;. Alternatively or optionally, the phone user 1115 actions can also encompass other information to improve the performance of the listing program 1200. For example, the phone user 1115 actions can include actual selection of the called phone numbers &lgr;x from the favorite phone number list &agr;s. With this information, the intuition module 1215 can, e.g., remove phone numbers &agr;x that have not been selected by the phone user 1115, but are nonetheless on the favorite phone number list 1120. Presumably, in these cases, the phone user 1115 prefers to dial this particular phone number &lgr;x using the number keys 1145 and feels he or she does not need to select it, e.g., if the phone number is well known to the phone user 1115. Thus, the corresponding listed phone number &agr;x will be replaced on the favorite phone number list &agr;s with another phone number &agr;x.
[0480] As another example, the phone user 1115 actions can include phone numbers from phone calls received by the mobile phone 1100 (i.e., incoming phone calls), which presumably correlate with the phone user's 1115 calling patterns to the extent that the phone number that is received represents a phone number that will likely be called in the future. In this case, the listing program 1200 may treat the received phone number similar to the manner in which it treats a called phone number &lgr;x, e.g., the outcome evaluation module 1230 determines whether the received phone number is found on the comprehensive phone number list &agr; and/or the selected phone number subset &agr;s, and the intuition module 1215 accordingly modifies the phone number probability distribution p based on this determination. Alternatively, a separate comprehensive phone number list can be maintained for the received phone numbers, so that a separate favorite phone number list associated with received phone numbers can be displayed to the user.
[0481] As still another example, the outcome value &bgr; can be time-based in that the cumulative time of a specific phone call (either incoming or outgoing) can be measured to determine the quality of the phone call, assuming that the importance of a phone call is proportional to its length. If the case of a relatively lengthy phone call, the intuition module 1215 can assign a probability value (if not found on the comprehensive phone number list &agr;) or increase the probability value (if found on the comprehensive phone number list &agr;) of the corresponding phone number higher than would otherwise be assigned or increased. In contrast, in the case of a relatively short phone call, the intuition module 1215 can assign a probability value (if not found on the comprehensive phone number list &agr;) or increase the probability value (if found on the comprehensive phone number list &agr;) of the corresponding phone number lower than would otherwise be assigned or increased. When measuring the quality of the phone call, the processing can be performed after the phone call is terminated.
[0482] Having now described the structure of the listing program 1200, the steps performed by the listing program 1200 will be described with reference to FIG. 20. In this process, the intuition module 1215 does not distinguish between phone numbers &agr;x that are listed in the phone number subset &agr;s and those that are found on the remainder of the comprehensive phone number list a.
[0483] First, the outcome evaluation module 1230 determines whether a phone number &lgr;x has been called (step 1305). Alternatively or optionally, the evaluation module 1230 may also determine whether a phone number &lgr;x has been received. If a phone number &lgr;x has not been received, the program 1200 goes back to step 1305. If a phone number &lgr;x has been called and/or received, the outcome evaluation module 1230 determines whether it is on the comprehensive phone number list &agr; and generates an outcome value &lgr; in response thereto (step 1315). If so (&bgr;=1), the intuition module 1215 directs the probability update module 1220 to update the phone number probability distribution p using a learning methodology to increase the probability value piix corresponding to the listed phone number &agr;x (step 1325). If not (&bgr;=0), the intuition module 1215 generates a corresponding phone number &agr;x and assigns a probability value px to it, in effect, adding it to the comprehensive phone number list &agr; (step 1330).
[0484] The phone number selection module 1225 then reorders the comprehensive phone number list &agr;x and selects the phone number subset &agr;x therefrom, and in this case, the listed phone numbers &agr;x with the highest probability values px (e.g., the top six) (step 1340). The phone number subset &agr;x is then displayed to the phone user 1115 as the favorite phone number list 1120 (step 1345). The listing program 1200 then returns to step 1305, where it is determined again if phone number &lgr;x has been called and/or received.
[0485] Referring to FIG. 21, the operation of the listing program 1200 will be described, wherein the intuition module 1215 does distinguish between phone numbers &agr;x that are listed in the phone number subset &agr;x and those that are found on the remainder of the comprehensive phone number list &agr;.
[0486] First, the outcome evaluation module 1230 determines whether a phone number &lgr;x has been called and/or received (step 1405). If a phone number &lgr;x has been called and/or received, the outcome evaluation module 1230 determines whether it matches a phone number &agr;x in either of the phone number subset &agr;s (in effect, the favorite phone number list 1120) or the comprehensive phone number list &agr; and generates an outcome value &bgr; in response thereto (steps 1415 and 1420). If the phone number &lgr;x, matches a phone number &agr;x on the favorite phone number list &agr;s (&bgr;=1), the intuition module 1215 directs the probability update module 1220 to update the phone number probability distribution p (or phone number probability sub-distributions p1 and p2) using a learning methodology to increase the probability value px corresponding to the listed phone number &agr;x (step 1425). If the called phone number &lgr;x does not match a phone number &agr;x on the comprehensive phone number list (&bgr;=0), the intuition module 1215 generates a corresponding phone number &agr;x and assigns a probability value px to it, in effect, adding it to the comprehensive phone number list &agr; (step 1430). If the called phone number &lgr;x does not match a phone number &agr;x on the favorite phone number list &agr;s, but matches one on the comprehensive phone number list &agr; (&bgr;=2), the intuition module 1215 assigns a probability value px to the already listed phone number &agr;x to, e.g., place the listed phone number &agr;x within or near the favorite phone number list &agr;s (step 1435).
[0487] The phone number selection module 1225 then reorders the comprehensive phone number list &agr;, and selects the phone number subset &agr;s therefrom, and in this case, the listed phone numbers &agr;x with the highest probability values px (e.g., the top six) (step 1440). The phone number subset &agr;s is then displayed to the phone user 1115 as the favorite phone number list 1120 (step 1445). The listing program 1200 then returns to step 1405, where it is determined again if phone number &lgr;x has been called and/or received.
[0488] Referring to FIG. 22, the operation of the listing program 1200 will be described, wherein the intuition module 1215 distinguishes between weekday and weekend phone calls.
[0489] First, the outcome evaluation module 1230 determines whether a phone number &lgr;x has been called (step 1505). Alternatively or optionally, the evaluation module 1230 may also determine whether a phone number &lgr;x has been received. If a phone number &lgr;x has not been received, the program 1200 goes back to step 1505. If a phone number &lgr;x has been called and/or received, the intuition module 1215 determines whether the current day is a weekend day or a weekend (step 1510). If the current day is a weekday, the weekday comprehensive phone list &agr;1 is operated on in steps 1515(1)-1545(1) in a similar manner as the comprehensive phone list &agr; is operated on in steps 1415-1440 in FIG. 21. In this manner, a favorite phone number list 1120 customized to weekday calling patterns is displayed to the phone user 1115. If the current day is a weekend day, the weekend comprehensive phone list &agr;2 is operated on in steps 1515(2)-1545(2) in a similar manner as the comprehensive phone list &agr; is operated on in steps 1415-1440 in FIG. 21. In this manner, a favorite phone number list 1120 customized to weekend calling patterns is displayed to the phone user 11 15. Optionally, rather than automatically customizing the favorite phone number list 1120 to the weekday or weekend for display to the phone user 1115, the phone user 1115 can select which customized favorite phone number list 1120 will be displayed. The listing program 1200 then returns to step 1505, where it is determined again if phone number &lgr;x has been called and/or received.
[0490] More specific details on the above-described operation of the mobile phone 1100 can be found in the Computer Program Listing Appendix attached hereto and previously incorporated herein by reference. It is noted that the file “Intuition Intelligence-mobilephone-outgoing.doc” generates a favorite phone number list only for outgoing phone calls, that is, phone calls made by the mobile phone. It does not distinguish between the favorite phone number list and the remaining phone numbers on the comprehensive list when generating outcome values, but does distinguish between weekday phone calls and weekend phone calls. The file “Intuition Intelligence-mobilephone-incoming.doc” generates a favorite phone number list only for incoming phone calls; that is, phone calls received by the mobile phone. It does not distinguish between the favorite phone number list and the remaining phone numbers on the comprehensive list when generating outcome values, and does not distinguish between weekday phone calls and weekend phone calls.
[0491] It should be noted that the files “Intuition Intelligence-mobilephone-outgoing.doc” and “Intuition Intelligence-mobilephone-incoming.doc” simulation programs were designed to emulate real-world scenarios and to demonstrate the learning capability of the priority listing program. To this end, the software simulation is performed on a personal computer with Linux Operating System Mandrake Version 8.2. This operating system was selected because the MySQL database, PHP and Apache Web Server are natively built in. The MySQL database acts as a repository and stores the call logs and tables utilized in the programs. The MySQL database is a very fast, multi-user relational database management system that is used for storing and retrieving information. The PHP is a cross-platform, Hyper Text Markup Language (HTML)-embedded, server-side, web scripting language to provide and process dynamic content. The Apache Web Server is a public-domain web server that receives a request, processes a request, and sends the response back to the requesting entity. Because a phone simulator was not immediately available, the phone call simulation was performed using a PyWeb Deckit Wireless Application Protocol (WAP) simulator, which is a front-end tool/browser that emulates the mobile phone, and is used to display wireless language content debug the code. It is basically a browser for handheld devices. The Deckit transcoding technology is built-in to allow one to test and design the WAP site offline. The transcoding is processed locally on the personal computer.
[0492] Single-User Television Channel Listing Program
[0493] The afore-described listing programs can be used for other applications besides prioritizing and anticipating watched television channels on a telephone. For example, referring to FIG. 23, a priority listing program 1700 (shown in FIG. 25) developed in accordance with the present inventions is described in the context of a television remote control 1600. The remote control 1600 comprises a keypad 1620 through which a remote control user 1615 (shown in FIG. 25) can remotely control a television (not shown), and which contains standard keys, such as number keys 1625, a channel up and down key 1630, a volume up and down key 1632, a scroll/selection keys 1635, and various other function keys. Referring further to FIG. 24, the remote control 1600 further includes keypad circuitry 1640, control circuitry 1645, memory 1650, a transmitter 1655, and an infrared (IR) emitter (or alternatively a light emitting diode (LED)) 1660. The keypad circuitry 1640 decodes the signals from the keypad 1620, as entered by the remote control user 1615, and supplies them to the control circuitry 1645. The control circuitry 1645 then provides the decoded signals to the transmitter 1655, which wirelessly transmits the signals to the television through the IR emitter 1660. The memory 1650 stores programs that are executed by the control circuitry 1645 for basic functioning of the remote control 1600. In many respects, these elements are standard in the industry, and therefore their general structure and operation will not be discussed in detail for purposes of brevity.
[0494] In addition to the standard features that typical remote controls have, however, the keypad 1620 contains a favorite channel key 1665 referred to as a “MYFAV” key. Much like the channel up or down keys 1630, operation of the favorite channel key 1665 immediately tunes (or switches) the television from the current television channel to the next television channel. Repetitive operation of the favorite channel key 1665 will switch the television from this new current television channel to the next one, and so on. Unlike the channel up or down keys 1630, however, the next television channel will not necessarily be the channel immediately above or below the current channel, but will tend to be one of the favorite television channels of the remote control user 1615.
[0495] It should be noted that rather than immediately and automatically switching television channels to a favorite television channel, operation of the favorite channel key 1665 can cause favorite television channel lists to be displayed on the television, similar to the previously described favorite phone number lists that were displayed on the mobile phone 1100. These lists will contain television channels that correspond to the remote control user 1615 favorite television channels, as determined by the remote control 1600. Once displayed on the television, the user can use the scroll/selection key 1635 on the keypad 1620 to select a desired channel from the favorite television channel list.
[0496] In any event, the priority listing program 1700, which is stored in the memory 1650 and executed by the control circuitry 1645, dynamically updates a comprehensive television channel list (described in further detail below) from which the next television channel will be selected. Preferably, the first channel on the comprehensive television channel list will be selected, then the second channel, then the third channel, and so on. The program 1700 updates the comprehensive television channel list based on the user's 1615 television watching pattern. For example, the program 1700 may maintain the comprehensive television channel list based on the number of times a television channel has been watched and the recent activity of the television channel, such that the comprehensive television channel list will likely contain a television channel that the remote control user 1615 is anticipated to watch at any given time. For example, if channels 2, 4, 6, and 7 have recently been watched numerous times, the program 1700 will tend to maintain these channels at the top of the comprehensive television channel list, so that they will be selected when the remote control user 1615 operates the favorite television channel key 1665.
[0497] To further improve the accuracy of anticipating the next channel that will be watched by the remote control user 1615, the program 1700 may optionally maintain several comprehensive television channel lists based on temporal information, such as, e.g., the day of the week (weekend or weekday) and/or time of day (day or evening). For example, a user 1615 may tend to watch a specific set of channels (e.g., 2, 4, and 8) between 8 pm and 10 pm on weekdays, and other set of channels (2, 5, and 11) between 3 pm and 6 pm on weekends. Or a user 1615 may tend to watch news programs between 10 pm and 12 pm on weekdays, and cartoons between 10 am and 12 pm on weekends. Thus, to further refine the process, the comprehensive television channel list can be divided into sublists that are selected and applied based on the current day of the week and/or time of the day.
[0498] To ensure that television channels that are quickly switched are not registered as being watched, the program 1700 only assumes that a program is watched if the remote control user 1615 has continuously watched the television channel for more than a specified period of time (e.g., five minutes). Thus, a television channel will only affect the comprehensive television channel list if this period of time is exceeded. This period of time can be fixed for all lengths of television programs, or optionally, can be based on the length of the television program (e.g., the longer the television program, the longer the time period). Optionally, programming information contained in a device, such as, e.g., a set top box or a video cassette recorder, can be used to determine if a television program is actually watched or not.
[0499] It should also be noted that although only a single user is illustrated, multiple users can obviously use the remote control 1600. In this case, usage of the remote control 1600 by multiple users will be transparent to the program 1700, which will maintain the comprehensive television channel list as if a single user was always using the remote control 1600. As will be described in further detail below, the program can be modified to maintain a television channel list for each of the users 1615, so that the television channel patterns of one user do not dilute or interfere with the television channel patterns of another user. In this manner, the comprehensive television channel list can be customized to the particular user that is currently operating the remote control 1600.
[0500] As will be described in further detail below, the listing program 1700 uses the existence or non-existence of a watched television channel on a comprehensive television channel list as a performance index &phgr; in measuring its performance in relation to its objective of ensuring that the comprehensive channel list will include the future watched television channel, so that the remote control user 1615 is not required to “surf” through all of the television channels or manually punch in the television channel using the number keys. In this regard, it can be said that the performance index &phgr; is instantaneous. Alternatively or optionally, the listing program 1700 can also use the location of the television channel on the comprehensive channel list as a performance index &phgr;.
[0501] Referring now to FIG. 25, the listing program 1700 includes a probabilistic learning module 1710 and an intuition module 1715, which are specifically tailored for the remote control 1600. The probabilistic learning module 1710 comprises a probability update module 1720, a television channel selection module 1725, and an outcome evaluation module 1730. Specifically, the probability update module 1720 is mainly responsible for learning the remote control user's 1615 television watching habits and updating the previously described comprehensive television channel list &agr; that places television channels in the order that they are likely to be watched in the future during any given time period. The outcome evaluation module 1730 is responsible for evaluating the comprehensive channel list &agr; relative to current television channels &lgr;x, watched by the remote control user 1615. The channel selection module 1725 is mainly responsible for selecting a television channel from the comprehensive channel list &agr; upon operation of the favorite television channel key 1665. Preferably, this is accomplished by selecting the channel at the top of the comprehensive channel list &agr;, then the second channel, third channel, and so on, as the favorite television channel key 1665 is repeatedly operated. The intuition module 1715 is responsible for directing the learning of the listing program 1700 towards the objective of selecting the television channel that is likely to be the remote control user's 1615 next watched television channel. In this case, the intuition module 1715 operates on the probability update module 1720, the details of which will be described in further detail below.
[0502] To this end, the channel selection module 1725 is configured to receive a television channel probability distribution p from the probability update module 1720, which is similar to equation [1] and can be represented by the following equation:
p(k)=[p1(k),p2(k),p3(k) . . . pn(k)],
[0503] where px is the probability value assigned to a specific television channel &agr;x; n is the number of television channels &agr;x on the comprehensive channel list &agr;, and k is the incremental time at which the television channel probability distribution was updated. Based on the television channel probability distributionp, the channel selection module 1725 generates the comprehensive channel list &agr;, which contains the listed television channels &agr;x ordered in accordance with their associated probability values px. For example, the first listed television channel &agr;x will be associated with the highest probability value px, while the last listed television channel &agr;x will be associated with the lowest probability value px. Thus, the comprehensive channel list &agr; contains all television channels ever watched by the remote control user 1615 and is unlimited. Optionally, the comprehensive channel list &agr; can contain a limited amount of television channels &agr;x, e.g., 10, so that the memory 1650 is not overwhelmed by seldom watched television channels, which may eventually drop off of the comprehensive channel list &agr;.
[0504] It should be noted that a comprehensive television channel list &agr; need not be separate from the television channel probability distribution p, but rather the television channel probability distribution p can be used as the comprehensive channel list &agr; to the extent that it contains a comprehensive list of television channels &agr;x matching all of the watched television channels &lgr;x. However, it is conceptually easier to explain the aspects of the listing program 1700 in the context of a comprehensive television channel list that is ordered in accordance with the corresponding probability values px, rather than in accordance with the order in which they are listed in the television channel probability distribution p.
[0505] From the comprehensive channel list &agr;, the channel selection module 1725 selects the television channel &agr;x that the television will be switched to. In the preferred embodiment, the selected television channel &agr;x will be that which corresponds to the highest probability value px, i.e., the top television channel &agr;x on the comprehensive channel list &agr;. The channel selection module 1725 will then select the next television channel &agr;x that the television will be switched to, which preferably corresponds to the next highest probability value px, i.e., the second television channel &agr;x on the comprehensive channel list &agr;, and so on. As will be described in further detail below, this selection process can be facilitated by using a channel list pointer, which is incremented after each channel is selected, and reset to “1” (so that it points to the top channel) after a television channel has been deemed to be watched or after the last channel on the comprehensive channel list &agr; has been reached.
[0506] As an example, consider Table 10, which sets forth in exemplary comprehensive television channel list &agr; with associated probability values px. 10 TABLE 10 Exemplary Probability Values for Comprehensive Television Channel List Number Listed Television Channels (&agr;1) Probability Values (p1) 1 2 0.286 2 11 0.254 3 4 0.142 4 26 0.114 5 35 0.097 6 9 0.054 7 48 0.033 8 76 0.012 9 5 0.008 10 15 0.003
[0507] In this exemplary case, channel 2, then channel 11, then channel 4, and so on, will be selected as the television channels to which the television will be sequentially switched. Optionally, these channels can selected as a favorite television channel list to be displayed on the television, since they are associated with the top three probability values px.
[0508] The outcome evaluation module 1730 is configured to receive a watched television channel &lgr;x from the remote control user 1615 via the keypad 1620 of the remote control 1600. For example, the remote control user 1615 can switch the television to the television channel &lgr;x using the number keys 1625 or channel-up or channel-down keys 1630 on the keypad 1620, operating the favorite channel key 1665 on the keypad 1620, or through any other means, including voice activation. In this embodiment, the television channel &lgr;x can be selected from a complete set of television channels &lgr;, i.e., all valid television channels that can be watched on the television. As previously discussed, the switched television channel will be considered to be a watched television channel only after a certain period of time has elapsed while the television is on that television channel. The outcome evaluation module 1730 is further configured to determine and output an outcome value &bgr; that indicates if the currently watched television channel &lgr;x matches a television channel &agr;x on the comprehensive channel list &agr;. In the illustrated embodiment, the outcome value &bgr; equals one of two predetermined values: “1” if the currently watched television channel &lgr;x matches a television channel &agr;x on the comprehensive channel list &agr;, and “0” if the currently watched television channel &lgr;x does not match a television channel &agr;x on the comprehensive channel list &agr;.
[0509] It can be appreciated that unlike in the duck game 300 where the outcome value &bgr; is partially based on the selected game move &agr;x, the outcome value &bgr; is technically not based on the listed television channel &agr;x selected by the channel selection module 1725, but rather whether a watched television channel &lgr;x matches a television channel &agr;x on the comprehensive channel list &agr; irrespective of whether it is the selected television channel. It should be noted, however, that the outcome value &bgr; can optionally or alternatively be partially based on a selected television channel.
[0510] The intuition module 1715 is configured to receive the outcome value &bgr; from the outcome evaluation module 1730 and modify the probability update module 1720, and specifically, the television channel probability distribution p, based thereon. Specifically, if the outcome value &bgr; equals “0,” indicating that the currently watched television channel &lgr;x does not match a television channel &agr;x on the comprehensive channel list &agr;x the intuition module 1715 adds the watched television channel &lgr;x to the comprehensive channel list &agr; as a listed television channel &agr;x.
[0511] The television channel &agr;x can be added to the comprehensive channel list &agr; in a variety of ways, including in the manner used by the program 1700 to add a telephone number in the mobile phone 1100. Specifically, the location of the added television channel &agr;x on the comprehensive channel list &agr; depends on the probability value px assigned or some function of the probability value px assigned.
[0512] For example, in the case where the number of television channels &agr;x is not limited, or the number of television channels &agr;x has not reached its limit, the television channel &agr;x may be added by assigning a probability value px to it and renormalizing the television channel probability distribution p in accordance with the following equations:
px(k+1)=f(x); [27]
pj(k+1)=pj(k)(1−f(x)); j≠i [∞]
[0513] where i is the added index corresponding to the newly added television channel &agr;x, px is the probability value corresponding to television channel &agr;x added to the comprehensive channel list &agr;, f(x) is the probability value px assigned to the newly added television channel &agr;x, pj is each probability value corresponding to the remaining television channels &agr;j on the comprehensive channel list &agr;, and k is the incremental time at which the television channel probability distribution was updated.
[0514] In the illustrated embodiment, the probability value px assigned to the added television channel &agr;x is simply the inverse of the number of television channels &agr;x on the comprehensive channel list &agr;, and thus f(x) equals 1/(n+1), where n is the number of television channels on the comprehensive channel list &agr; prior to adding the television channel &agr;x. Thus, equations [27] and [28] break down to: 28 p i ( k + 1 ) = 1 n + 1 ; [27-1] p j ( k + 1 ) = p j ( k ) n n + 1 ; j ≠ i [28-1]
[0515] In the case where the number of television channels &agr;x is limited and has reached its limit, the television channel &agr; with the lowest corresponding priority value px is replaced with the newly watched television channel &lgr;x by assigning a probability value px to it and renormalizing the television channel probability distribution p in accordance with the following equations:
pi(k+1)=f(x); [29]
[0516] 29 p j ( k + 1 ) = p j ( k ) ∑ j ≠ i n p j ( k ) ( 1 - f ( x ) ) ; j ≠ i [ 30 ]
[0517] where i is the index used by the removed television channel &agr;x, px is the probability value corresponding to television channel &agr;x added to the comprehensive channel list &agr;,f(x) is the probability value pm assigned to the newly added television channel &agr;, f(x) is each probability value corresponding to the remaining television channels &agr;j on the comprehensive channel list &agr;, and k is the incremental time at which the television channel probability distribution was updated.
[0518] As previously stated, in the illustrated embodiment, the probability value px assigned to the added television channel &agr;x is simply the inverse of the number of television channels &agr;x on the comprehensive channel list &agr;, and thus f(x) equals 1/n, where n is the number of television channels on the comprehensive channel list &agr;. Thus, equations [35] and [36] break down to: 30 p i ( k + 1 ) = 1 n ; [29-1] p j ( k + 1 ) = p j ( k ) ∑ j ≠ i n p j ( k ) ( n - 1 n ) ; j ≠ i [30-1]
[0519] It should be appreciated that the speed in which the automaton learns can be controlled by adding the television channel &agr;x to specific locations within the television channel probability distribution p. For example, the probability value px assigned to the added television channel &agr;x, can be calculated as the mean of the current probability values px, such that the television channel &agr;i will be added to the middle of the comprehensive channel list &agr; to effect an average learning speed. The probability value px assigned to the added television channel &agr;x can be calculated as an upper percentile (e.g. 25%) to effect a relatively quick learning speed. Or the probability value pi assigned to the added television channel &agr;x can be calculated as a lower percentile (e.g. 75%) to effect a relatively slow learning speed. It should be noted that if there is a limited number of television channels &agr;x on the comprehensive channel list &agr;, thereby placing the lowest television channels &agr;x in the likelihood position of being deleted from the comprehensive channel list &agr;x the assigned probability value px should be not be so low as to cause the added television channel &agr;x to oscillate on and off of the comprehensive channel list &agr; when it is alternately watched and not watched.
[0520] In any event, if the outcome value &bgr; received from the outcome evaluation module 1730 equals “1,” indicating that the currently watched television channel &lgr;x matches a television channel &agr;x on the comprehensive channel list &agr;, the intuition module 1715 directs the probability update module 1720 to update the television channel probability distribution p using a learning methodology. In the illustrated embodiment, the probability update module 1720 utilizes a linear reward-inaction P-type update.
[0521] As an example, assume that a currently watched television channel &lgr;x matches a television channel &agr;5 on the comprehensive channel list &agr;, thus creating an outcome value &bgr;=1. Assume also that the comprehensive channel list &agr; currently contains 10 television channels &agr;x. In this case, general updating equations [6] and [7] can be expanded using equations [10] and [11], as follows: 31 p 5 ( k + 1 ) = p 5 ( k ) + ∑ j = 1 j ≠ 5 10 ap j ( k ) ;
p1(k+1)=p1(k)−ap1(k);
p2(k+1)=p2(k)−ap2(k);
p3(k+1)=p3(k)−ap3(k);
p4(k+1)=p4(k)−ap4(k);
p6(k+1)=p6(k)−ap6(k);
p7(k+1)=p7(k)−ap7(k);
p8(k+1)=p8(k)−ap8(k);
p9(k+1)=p9(k)−ap9(k);
p10(k+1)=p10(k)−ap10(k)
[0522] Thus, the corresponding probability value p5 is increased, and the television channel probability values px corresponding to the remaining television channels &agr;x are decreased. The value of a is selected based on the desired learning speed. The lower the value of &agr;, the slower the learning speed, and the higher the value of &agr;, the higher the learning speed. In the preferred embodiment, the value of a has been chosen to be 0.03. It should be noted that the penalty updating equations [8] and [9] will not be used, since in this case, a reward-penalty P-type update is not used.
[0523] Thus, it can be appreciated that, in general, the more a specific listed television channel &agr;x is watched relative to other listed television channels &agr;x, the more the corresponding probability value px is increased, and thus the higher that listed television channel &agr;x is moved up on the comprehensive channel list &agr;. As such, the chances that the listed television channel &agr;x will be selected will be increased. In contrast, the less a specific listed television channel &agr;x is watched relative to other listed television channels &agr;x, the more the corresponding probability value px is decreased (by virtue of the increased probability values px corresponding to the more frequently watched listed television channels &agr;x), and thus the lower that listed television channel &agr;x is moved down on the comprehensive channel list &agr;. As such, the chances that the listed television channel &agr;x will be selected by the channel selection module 1725 will be decreased.
[0524] It can also be appreciated that due to the nature of the learning automaton, the relative movement of a particular listed television channel &agr;x is not a matter of how many times the television channel &agr;x is watched, and thus, the fact that the total number of times that a particular listed television channel &agr;x has been watched is high does not ensure that it will be selected. In reality, the relative placement of a particular listed television channel &agr;x on the comprehensive channel list &agr;s is more of a function of the number of times that the listed television channel &agr;x has been recently watched. For example, if the total number of times a listed television channel &agr;i is watched is high, but has not been watched in the recent past, the listed television channel &agr;x may be relatively low on the comprehensive channel list &agr; and thus it may not be selected. In contrast, if the total number of times a listed television channel &agr;x is watched is low, but it has been watched in the recent past, the listed television channel &agr;x may be relatively high on the comprehensive channel list &agr;x and thus it may be selected. As such, it can be appreciated that the learning automaton quickly adapts to the changing watching patterns of a particular remote control user 1615.
[0525] It should be noted, however, that a television channel probability distribution p can alternatively be purely based on the frequency of each of the television channels A. For example, given a total of n television channels watched, and a total number of times that each television channel is watched f1, f2, f3 . . . , the probability values px for the corresponding listed television channels &agr;x can be: 32 p i ( k + 1 ) = fi n [ 31 ]
[0526] Noteworthy, each probability value px is not a function of the previous probability value px (as characterized by learning automaton methodology), but rather the frequency of the listed television channel &agr;x and total number of watched television channels n. With the purely frequency-based learning methodology, when a new television channel &agr;x is added to the comprehensive channel list &agr;x its corresponding probability value px will simply be 1/n, or alternatively, some other function of the total number of watched television channels n. Optionally, the total number of watched television channels n is not absolute, but rather represents the total number of watched television channels n made in a specific time period, e.g., the last three months, last month, or last week. In other words, the television channel probability distribution p can be based on a moving average. This provides the frequency-based learning methodology with more dynamic characteristics.
[0527] In any event, as described above, a single comprehensive television channel list &agr; that contains all television channels watched regardless of the time and day of the week is generated and updated. Optionally, several comprehensive television channel lists a can be generated and updated based on the time and day of the week. For example, Tables 11 and 12 below set forth exemplary comprehensive television channel lists &agr;1 and &agr;2 that respectively contain television channels &agr;1x and &agr;2x that are watched during the weekdays and weekend. 11 TABLE 11 Exemplary Probability Values for Comprehensive Weekday Television Channel List Listed Weekday Number Television Channels (&agr;11) Probability Values (p1) 1 4 0.263 2 11 0.188 3 48 0.162 4 29 0.133 5 9 0.103 6 2 0.075 7 88 0.033 8 38 0.025 9 7 0.014 10 15 0.004
[0528] 12 TABLE 12 Exemplary Probability Values for Comprehensive Weekend Television Channel List Listed Weekend Number Television Channels (&agr;11) Probability Values (p1) 1 11 0.256 2 7 0.209 3 4 0.153 4 38 0.125 5 93 0.083 6 29 0.067 7 48 0.043 8 9 0.032 9 2 0.020 10 8 0.012
[0529] Notably, the top five locations of the exemplary comprehensive television channel lists &agr;1 and &agr;2 contain different television channels &agr;1x and &agr;2x, presumably because certain television channels &agr;1x (e.g., 48, 29, and 9) were mostly only watched during the weekdays, and certain television channels &agr;2x (e.g., 7, 38, and 93) were mostly only watched during the weekends. The top five locations of the exemplary comprehensive television channel lists &agr;1 and &agr;2 also contain common television channels &agr;x and &agr;2x, presumably because certain television channels &agr;1x and &agr;2x (e.g., 4 and 11) were watched during the weekdays and weekends. Notably, these common television channels &agr;1x and &agr;2x are differently ordered in the exemplary comprehensive television channel lists &agr;1 and &agr;2, presumably because the remote control user's 1615 weekday and weekend watching patterns have differently influenced the ordering of these television channels. Although not shown, the single comprehensive list &agr; can be subdivided, or the comprehensive channel lists &agr;1 and &agr;2 can be further subdivided, e.g., by day and evening.
[0530] When there are multiple comprehensive television channel lists a that are divided by day and/or time, the channel selection module 1725, outcome evaluation module 1730, probability update module 1720, and intuition module 1715 operate on the comprehensive channel lists a based on the current day and/or time (as obtained by a clock or calendar stored and maintained by the control circuitry 1645). Specifically, the intuition module 1715 selects the particular comprehensive list &agr; that will be operated on. For example, during a weekday, the intuition module 1715 will select the comprehensive channel lists &agr;1, and during the weekend, the intuition module 1715 will select the comprehensive channel lists &agr;2.
[0531] The channel selection module 1725 will maintain the ordering of all of the comprehensive channel lists &agr;, but will select the television channel from the particular comprehensive television channel list &agr; selected by the intuition module 1715. For example, during a weekday, the channel selection module 1725 will select the television channel from the comprehensive channel list &agr;1, and during the weekend, the channel selection module 1725 will select the television channel from the comprehensive channel list &agr;2. Thus, it can be appreciated that the particular television channel to which the television will be switched will be customized to the current day, thereby increasing the chances that the next television channel &lgr;x watched by the remote control user 1615 will be the selected television channel.
[0532] The outcome evaluation module 1730 will determine if the currently watched television channel &lgr;x matches a television channel &agr;x on the comprehensive channel list &agr; selected by the intuition module 1715 and generate an outcome value &bgr; based thereon, and the intuition module 1715 will accordingly modify the television channel probability distribution p corresponding to the selected comprehensive television channel list &agr;. For example, during a weekday, the outcome evaluation module 1730 determines if the currently watched television channel &lgr;x matches a television channel &agr;x on the comprehensive channel list &agr;1, and the intuition module 1715 will then modify the television channel probability distribution p corresponding to the comprehensive channel list &agr;1. During a weekend, the outcome evaluation module 1730 determines if the currently watched television channel &lgr;x matches a television channel &agr;x on the comprehensive channel list &agr;2, and the intuition module 1715 will then modify the television channel probability distribution p corresponding to the comprehensive channel list &agr;2.
[0533] The intuition module 1715 may also prevent any one probability value px from overwhelming the remaining probability values px by limiting it to a particular value, e.g., 0.5. In this sense, the learning module 1710 will not converge to any particular probability value px, which is not the objective of the remote control 1600. That is, the objective is not to find a single favorite television channel, but rather a list of favorite television channels that dynamically changes with the remote control user's 1615 changing watching patterns. Convergence to a single probability value px would defeat this objective.
[0534] So far, it has been explained that the listing program 1700 uses the instantaneous outcome value &bgr; as a performance index &phgr; in measuring its performance in relation to its objective of select a television channel that will be watched by the remote control user 1615. It should be appreciated, however, that the performance of the listing program 1700 can also be based on a cumulative performance index &phgr;. For example, the listing program 1700 can keep track of a percentage of the watched television channels &lgr;x that match a television channel &agr;x on the comprehensive channel list &agr; or portion thereof, or a consecutive number of watched television channels &lgr;x that do not match a television channel &agr;x on the comprehensive channel list &agr; or portion thereof, based on the outcome value &bgr;. Based on this cumulative performance index &phgr;, the intuition module 1715 can modify the learning speed or nature of the learning module 1710.
[0535] Optionally, the outcome value &bgr; can be time-based in that the cumulative time that a television channel is watched can be measured to determine the quality of the watched television channel. If the case of a relatively lengthy time the television channel is watched, the intuition module 1715 can assign a probability value (if not found on the comprehensive channel list &agr;) or increase the probability value (if found on the comprehensive channel list &agr;) of the corresponding television channel higher than would otherwise be assigned or increased. In contrast, in the case of a relatively short time the television channel is watched, the intuition module 1715 can assign a probability value (if not found on the comprehensive channel list &agr;) or increase the probability value (if found on the comprehensive channel list &agr;) of the corresponding television channel lower than would otherwise be assigned or increased. When measuring the quality of the watched television channel, the processing can be performed after the television channel is switched.
[0536] It should be noted that, in the case where a comprehensive television channel list is displayed on the screen of the television for selection by the remote control user 1615, the channel selection module 1725 may optionally select a television channel subset from the comprehensive channel list &agr; for eventual display to the remote control user 1615 as a comprehensive television channel list. Updating of a comprehensive television channel list that contains a television channel subset, and selection of the comprehensive television channel list for display, is similar to that accomplished in the previously described mobile phone 1100 when updating the comprehensive phone number list and selecting the favorite phone number therefrom.
[0537] Although the program 1700 is described as being stored within a remote control 1600, it can be distributed amongst several components within a remote control television system, or another component within the remote control television system, e.g., within the television, itself, or some other device associated with the television, e.g., a cable box, set top box, or video cassette recorder. In addition, although the program 1700 is described for use with a television, it should be noted that it can be applied to other consumer electronic equipment on which users can watch or listen to programs by switching channels, e.g., stereo equipment, satellite radio, MP3 player, Web devices, etc.
[0538] Having now described the structure of the listing program 1700, the steps performed by the listing program 1700 will be described with reference to FIG. 26. First, the outcome evaluation module 1730 determines whether a television channel &lgr;x has been newly watched (step 1805). As previously discussed, this occurs when a predetermined period of time has elapsed while the television is tuned to the television channel. If a television channel &lgr;x has been newly watched, the outcome evaluation module 1730 determines whether it matches a television channel &agr;x on the comprehensive channel list &agr; and generates an outcome value &bgr; in response thereto (step 1815). If so (&bgr;=1), the intuition module 1715 directs the probability update module 1720 to update the television channel probability distribution p using a learning methodology to increase the probability value px corresponding to the listed television channel &agr;x (step 1825). If not (&bgr;=0), the intuition module 1715 generates a corresponding television channel &agr;x and assigns a probability value px to it, in effect, adding it to the comprehensive channel list &agr; (step 1830). The channel selection module 1725 then reorders the comprehensive channel list &agr; (step 1835), sets the channel list pointer to “1” (step 1840), and returns to step 1805.
[0539] If a television channel &lgr;x has not been newly watched at step 1805, e.g., if the predetermined period of time has not expired, the channel selection module 1725 determines whether the favorite channel key 1665 has been operated (step 1845). If so, the channel selection module 1725 selects a listed television channel &agr;x, and in this case, the listed television channel &agr;i corresponding to the channel list pointer (step 1850). The television is then switched to the selected television channel &agr;x (step 1855), and the channel list pointer is incremented (step 1860). After step 1860, or if the favorite channel key 1665 has not been operated at step 1845, the listing program 1700 then returns to step 1805, where it is determined again if television channel &lgr;x has been watched.
[0540] Referring now to FIG. 27, another priority listing program 2000 (shown in FIG. 28) developed in accordance with the present inventions is described in the context of another television remote control 1900. The remote control 1900 is similar to the previously described remote control 1600 with the exception that it comprises a keypad 1920 that alternatively or optionally contains a specialized favorite channel key 1965 referred to as a “LINKFAV” key. The specialized favorite channel key 1965 is similar to the generalized favorite channel key 1965 in that its operation immediately and automatically switches the television from the current television channel to a next television channel that tends to correspond to one of the user's 1615 favorite television channels. Unlike with the generalized favorite channel key 1965, however, the next television channel will tend to be one of the user's 1615 favorite television channels based on the specific (as opposed to a general) channel watching pattern that the remote control user 1615 is currently in.
[0541] To this end, the program 2000 dynamically updates a plurality of linked comprehensive television channel lists from which the next television channel will be selected. Like with the generalized comprehensive television channel list, the program 2000 may maintain each of the linked comprehensive television channel lists based on the number of times a television channel has been watched, and the recent activity of the television channel. The linked comprehensive television channel lists are arranged and updated in such a manner that a selected one will be able to be matched and applied to the specific channel watching pattern that the remote control user 1615 is currently in.
[0542] Specifically, each linked comprehensive television channel list corresponds to a value of a specified television channel parameter, such that, when the remote control user 1615 operates the specialized favorite television key 1965, the linked comprehensive television channel list corresponding to the value exhibited by the currently watched television channel can be recalled, and thus, the next channel selected from that recalled list will be more likely to be the television channel that the remote control user 1615 desires to watch. A channel parameter can, e.g., include a switched channel number (in which case, the values may be 2, 3, 4, 5, etc.), channel type (in which case, the values may be entertainment, news, drama, sports, comedy, education, food, movies, science fiction, cartoon, action, music, shopping, home), channel age/gender (in which case, the values may be adult, teenage, kids, women, etc.), or channel rating (in which case, the values may be TV-Y, TV-Y7, TV-14, TV-MA, etc.). If the channel parameter is a channel type, channel age/gender or channel rating, a device (such as, e.g., a set top box, television or video cassette recorder) can be used to extract this information from the incoming program signal.
[0543] For example, if the channel parameter is a switched channel number, and if the television has been recently and often switched from channel 2 to channels 4, 8, and 11, or vice versa, the program 2000 will tend to maintain channels 4, 8, and 11 at the top of a list corresponding to channel 2, so that these favorite channels will be selected when the remote control user 1615 is currently watching channel 2 and operates the specialized favorite television channel key 1965. As another example, if the channel parameter is a channel type, and if movie channels 14 (TNT), 24 (MAX), and 26 (HBO3) have been recently watched numerous times, the program 2000 will tend to maintain these channels at the top of a list corresponding to movie channels, so that these favorite channels will be selected when the remote control user 1615 is currently watching a movie channel and operates the specialized favorite television channel key 1965.
[0544] As with the previously described program 1700, the program 2000 may optionally maintain the specialized television channel lists based on temporal information, such as, e.g., the day of the week (weekend or weekday) and/or time of day (day or evening). Thus, the specialized television channel lists can be further divided into sublists that are selected and applied based on the current day of the week and/or time of the day.
[0545] As with the program 1700, the program 2000 only assumes that a program is watched if the remote control user 1615 has continuously watched the television channel for more than a specified period of time (e.g., five minutes), so that a television channel will only affect the linked comprehensive television channel lists when this period of time is exceeded. Also, in the case where the television channel parameter is a switched channel number, selection of the next television channel from the specialized television channel lists, which would quickly vary with time, would be unstable without requiring a certain period of time to expire before a television channel can be considered watched. For example, without this feature, operation of the specialized favorite television channel key 1965 may switch the television from channel 2 to 4 if channel 4 is at the top of the linked comprehensive television channel list corresponding with channel 2, and then further operation of the specialized favorite television channel key 1965 may switch the television from channel 4 back to channel 2 if channel 2 is at the top of the linked comprehensive television channel list corresponding to channel 4. The channel would then switch back in and forth between channel 2 and 4 when the specialized favorite television channel key 1965 is further operated.
[0546] Thus, an assumption that a channel is a currently watched channel after a period of time has expired would prevent this adverse effect by forcing the program 2000 to select one linked comprehensive television channel list from which the unique channels can be sequentially selected. For example, when the currently watched television channel is channel 2, operation of the specialized favorite channel key 1965 may switch the television channel from channel 2 those channels that are on the linked comprehensive television channel list corresponding to channel 2. The predetermined period of time will, therefore, have to expire before the linked television channel, i.e., channel 2, is changed to the currently watched television channel.
[0547] As briefly discussed with respect to the program 1700, the program 2000 can be modified to maintain each of the specialized television channel lists for multiple users, so that the television channel patterns of one user do not dilute or interfere with the television channel patterns of another user. It should be noted, however, that in many cases, the specific channel watching patterns will be so unique to the users 1615 that the separate maintenance of the lists will not be necessary-at least with respect to the specialized favorite television channel key 1965. For example, a specific television channel pattern that is unique to kids (e.g., cartoons) will typically not conflict with a specific television channel pattern that is unique to adults (e.g., news).
[0548] As will be described in further detail below, the listing program 2000 uses the existence or non-existence of a watched television channel on the pertinent linked comprehensive television channel list as a performance index &phgr; in measuring its performance in relation to its objective of ensuring that the pertinent linked channel list will include the future watched television channel, so that the remote control user 1615 is not required to “surf” through all of the television channels or manually punch in the television channel using the number keys. In this regard, it can be said that the performance index &phgr; is instantaneous. Alternatively or optionally, the listing program 2000 can also use the location of the television channel in the pertinent linked comprehensive channel list as a performance index &phgr;.
[0549] Referring now to FIG. 28, the listing program 2000 includes a probabilistic learning module 2010 and an intuition module 2015, which are specifically tailored for the remote control 1900. The probabilistic learning module 2010 comprises a probability update module 2020, a television channel selection module 2025, and an outcome evaluation module 2030.
[0550] Specifically, the probability update module 2020 is mainly responsible for learning the remote control user's 1615 television watching habits and updating linked comprehensive television channel lists &agr;1-&agr;m that places television channels &agr;x in the order that they are likely to be watched in the future during any given time period. m equals the number of values associated with the pertinent television channel parameter. For example, if the television channel parameter is a channel number, and there are 100 channels, m equals 100. If the television channel parameter is a channel type, and there are ten channel types, m equals 10.
[0551] The outcome evaluation module 2030 is responsible for evaluating the linked comprehensive channel lists &agr;1-am relative to current television channels &lgr;x watched by the remote control user 1615. The channel selection module 2025 is mainly responsible for selecting a television channel &agr;x from the pertinent linked comprehensive channel list &agr;x upon operation of the favorite television channel key 1965. Preferably, this is accomplished by selecting the channel &agr;x at the top of the pertinent linked comprehensive channel list &agr;x then the second channel, third channel, and so on, as the specialized favorite channel key 1965 is repeatedly operated.
[0552] The intuition module 2015 is responsible for directing the learning of the listing program 2000 towards the objective of selecting the television channel &agr;x that is likely to be the remote control user's 1615 next watched television channel &agr;x. In this case, the intuition module 2015 selects the pertinent linked comprehensive channel list &agr;, and operates on the probability update module 2020, the details of which will be described in further detail below.
[0553] To this end, the channel selection module 2025 is configured to receive multiple television channel probability distributions p1-pm from the probability update module 2020, which is similar to equation [1] and can be represented by the following equation:
p1(k)=[p11(k),p12(k),p13(k) . . . p1n(k)];
p2(k)=[p21(k),p22(k),p23(k) . . . p2n(k)];
p3(k)=[p31(k),p32(k),p33(k) . . . p3n(k)]
pm(k)[pm1(k),pm2(k),pm3(k) . . . pmn(k)] [1-4]
[0554] where m is the number of probability distributions, i.e., the number of values associated with the pertinent television channel parameter; px is the probability value assigned to a specific television channel &agr;x, n is the number of television channels &agr;x on the comprehensive channel list &agr;, and k is the incremental time at which the television channel probability distribution was updated.
[0555] Based on the television channel probability distribution p1-pm, the channel selection module 2025 generates the linked comprehensive channel lists &agr;1-am, each of which contains the listed television channels &agr;x ordered in accordance with their associated probability values px. Thus, each linked comprehensive channel list &agr; contains all watched television channels &agr;x exhibiting a value corresponding to the list. For example, if the television channel parameter is a switched channel number, each linked comprehensive channel list &agr; will be linked with a channel number and will contain all television channels &agr;x ever watched by the remote control user 1615 that were switched to and from that television channel. If the television channel parameter is a channel type, each linked comprehensive channel list &agr; will be linked with a channel type and will contain all television channels &agr;x of that channel type ever watched by the remote control user 1615. As with the comprehensive channel list &agr; described with respect to the program 1700, each of the linked comprehensive channel lists &agr;1-&agr;m can be unlimited, or optionally, contain a limited amount of television channels &agr;x, e.g., 10, so that the memory 1650 is not overwhelmed by seldom watched television channels.
[0556] As with the previously described comprehensive television channel list &agr;x each of the linked comprehensive channel lists &agr;1-am need not be separate from their respective television channel probability distributions p1-pm, but rather a television channel probability distribution p can be used as a linked comprehensive channel list &agr; to the extent that it contains a comprehensive list of the linked television channels &agr;x.
[0557] From the linked comprehensive channel lists &agr;1-&agr;m, the channel selection module 2025 selects the list corresponding to the television channel parameter value exhibited by the current television channel watched, and then selects, from that list, a television channel &agr;x that the television will be switched to. In the preferred embodiment, the selected television channel &agr;x will be that which corresponds to the highest probability value px, i.e., the top television channel &agr;i in the selected linked comprehensive channel list &agr;. The channel selection module 2025 will then select the next television channel &agr;x that the television will be switched to, which preferably corresponds to the next highest probability value px, i.e., the second television channel &agr;x in the selected linked comprehensive channel list &agr;, and so on. As previously described above, this selection process can be facilitated by using a channel list pointer. In the preferred embodiment, once last television channel &agr;x is selected, the channel selection module 2025 will select the current channel that was watched prior to initiation of the selection process, and will then go through the selected linked comprehensive channel list &agr; again. Optionally, the channel selection module 2025 will only cycle through a subset of the selected linked comprehensive channel list &agr;, e.g., the top three.
[0558] As an example, consider Table 13, which sets forth exemplary linked comprehensive television channel lists &agr; with associated probability values px. In this case, the channel parameter is a switched channel number. 13 TABLE 13 Exemplary Probability Values for Linked Comprehensive Television Channel Lists Number Listed Television Channels (&agr;1) Probability Values (p1) Television Channel 2 1 11 0.310 2 26 0.283 3 4 0.202 4 9 0.093 5 35 0.057 6 95 0.022 7 39 0.012 8 5 0.011 9 7 0.007 10 38 0.003 Television Channel 4 1 2 0.280 2 8 0.238 3 9 0.168 4 38 0.119 5 30 0.084 6 83 0.032 7 5 0.028 8 7 0.018 9 33 0.009 10 93 0.004 . . . Television Channel 100 1 93 0.294 2 48 0.228 3 84 0.172 4 11 0.013 5 9 0.082 6 88 0.061 7 94 0.027 8 7 0.013 9 38 0.008 10 98 0.002
[0559] In this exemplary case, if the currently watched channel is channel 2, channel 11, then channel 26, then channel 4, and so on, will be selected as the television channels to which the television will be sequentially switched. If the currently watched channel is channel 4, channels 2, 8, and 9, and so on, will be selected. If the currently watched channel is channel 100, channels 93, 48, and 84 will be selected. Notably, there is no corresponding linked comprehensive television channel list &agr; for channel 3, presumably because channel 3 has never been watched.
[0560] As with the previously described outcome evaluation module 1730, the outcome evaluation module 2030 is configured to receive a watched television channel 4 from the remote control user 1615 via the keypad 1920 using any one of a variety of manners. The outcome evaluation module 2030 is further configured to determine and output an outcome value &bgr; that indicates if the currently watched television channel &lgr;x matches a television channel &agr;x on the linked comprehensive channel list &agr;x as selected by the intuition module 2015 described below. In the illustrated embodiment, the outcome value &bgr; equals one of two predetermined values: “1” if the currently watched television channel &lgr;x matches a television channel &agr;x on the selected linked comprehensive channel list &agr;x and “0” if the currently watched television channel &lgr;x does not match a television channel &agr;x on the selected linked comprehensive channel list &agr;.
[0561] The intuition module 2015 is configured to select the linked comprehensive channel list a corresponding to the television channel parameter value exhibited by the currently watched television channel &lgr;x. This selected linked comprehensive channel list &agr; is the list that is operated on by the outcome evaluation module 2030 described above. The intuition module 2015 is further configured to receive the outcome value &bgr; from the outcome evaluation module 2030 and modify the probability update module 2020, and specifically, the television channel probability distribution p corresponding to the selected linked comprehensive channel list &agr;. Specifically, if the outcome value &bgr; equals “0,” indicating that next watched television channel &lgr;x does not match a television channel &agr;x on the selected linked comprehensive channel list &agr;, the intuition module 2015 adds the watched television channel &lgr;x to the selected linked comprehensive channel list &agr; as a listed television channel &agr;x. The television channel &agr;x can be added to the selected linked comprehensive channel list &agr; in a manner similarly described with respect to the intuition module 1715. If the outcome value &bgr; received from the outcome evaluation module 2030 equals “1,” indicating that the next watched television channel &lgr;x matches a television channel &agr;x on the selected linked comprehensive channel list &agr;x the intuition module 2015 directs the probability update module 2020 to update the corresponding television channel probability distribution p in the manner previously described with respect to intuition module 1715.
[0562] Optionally, the intuition module 2015 can be configured to select the linked comprehensive channel list &agr; corresponding to the next watched television channel &lgr;x and update that based on whether the current watched television channel &lgr;x is found on that list-in effect, creating a bilateral link between the currently watched television channel &lgr;x and the next watched television channel &lgr;x rather than just a unilateral link from the currently watched television channel &lgr;x to the next watched television channel &lgr;x Thus, in this case, two linked comprehensive channel lists a will be updated for each television channel &lgr;x that is watched (one for the currently watched television channel &lgr;x , and one for the next watched television channel &lgr;x).
[0563] In the case where the channel selection module 2025 selects a subset of the selected linked comprehensive television channel list &agr; (e.g., for display to the remote control user 1615 as a favorite television channel list) or cycles through a subset of the linked comprehensive television channel list &agr;x the outcome evaluation module 2030 may generate more outcome values &bgr;. For example, in this case, the outcome evaluation module 2030 may generate an outcome value &bgr; equal to “1” if the currently watched television channel &lgr;x matches a television channel &agr;x in the previously selected television channel subset, “0” if the currently watched television channel &lgr;x does not match a television channel &agr;x on the selected linked comprehensive television channel list &agr;x and “2” if the currently watched television channel &lgr;x matches a television channel &agr;x on the selected linked comprehensive phone number list &agr;, but not in the previously selected television channel subset. If the outcome value is “0” or “1”, the intuition module 2015 will direct the probability update module 2020 as previously described. If the outcome value is “2”, however, the intuition module 2015 will not direct the probability update module 2020 to update the probability distribution p using a learning methodology, but instead will assign a probability value px to the listed television channel &agr;x. For example, the assigned probability value px may be higher than that corresponding to the last television channel &agr;x in the selected television channel subset, in effect, replacing that last television channel &agr;x with the listed television channel &agr;x corresponding to the currently watched television channel &lgr;x.
[0564] The program 2000 can include other optional features, such as those previously described with respect to the program 1700. For example, for each television channel, several linked comprehensive television channel lists &agr; can be generated and updated based on the time and day of the week. The intuition module 2015 may also prevent any one probability value px from overwhelming the remaining probability values px within each linked probability distribution p by limiting it to a particular value, e.g., 0.5. Also, the performance of the listing program 2000 can be based on a cumulative performance index &phgr; rather than an instantaneous performance index &phgr;. The outcome value &bgr; can be time-based in that the cumulative time that a television channel is watched can be measured to determine the quality of the watched television channel.
[0565] Having now described the structure of the listing program 2000, the steps performed by the listing program 2000 will be described with reference to FIG. 29. First, the outcome evaluation module 2030 determines whether a television channel &lgr;x has been newly watched (step 2105). As previously discussed, this occurs when a predetermined period of time has elapsed while the television is tuned to the television channel. If a television channel &lgr;x has been newly watched, the intuition module 2015 selects the linked comprehensive channel list &agr; corresponding to a television channel parameter value exhibited by the currently watched channel &lgr;x (step 2110). For example, if the television channel parameter is a switched channel number, and the currently watched channel &lgr;x is channel 2, the outcome evaluation module 2030 will select the linked comprehensive channel list &agr; corresponding to channel 2. If the television channel parameter is a channel type, and the currently watched channel &lgr;x is a sports channel, the outcome evaluation module 2030 will select the linked comprehensive channel list &agr; corresponding to sports.
[0566] The outcome evaluation module 2030 then determines whether the watched television channel &lgr;x matches a television channel &agr;i on the selected linked comprehensive channel list &agr; (step 2115). If so (&bgr;=1), the intuition module 2015 directs the probability update module 2020 to update the corresponding television channel probability distribution p using a learning methodology to increase the probability value px corresponding to the listed television channel &agr;x (step 2125). If not (&bgr;=0), the intuition module 2015 generates a corresponding television channel &agr;x and assigns a probability value px to it, in effect, adding it to the selected linked comprehensive channel list &agr; (step 2130). The channel selection module 2025 then reorders the selected linked comprehensive channel list &agr; (step 2135), sets the channel list pointer for the selected linked comprehensive channel list &agr; to “1” (step 2140), and returns to step 2105.
[0567] If a television channel &lgr;x has not been newly watched at step 2105, e.g., if the predetermined period of time has not expired, the channel selection module 2025 determines whether the favorite channel key 1965 has been operated (step 2145). If so, the channel selection module 2025 selects the linked comprehensive channel list &agr; corresponding to the television channel parameter value exhibited by the currently watched channel &lgr;x (step 2150), and then selects a listed television channel therefrom, and in this case, the listed television channel &agr;x corresponding to the channel list pointer for the selected linked comprehensive channel list &agr; (step 2155). The television is then switched to the selected television channel &agr;x (step 2160), and the channel list pointer for the selected linked comprehensive channel list &agr; is incremented (step 2165). After step 2165, or if the favorite channel key 1965 has not been operated at step 2145, the listing program 2000 then returns to step 2105, where it is determined again if a television channel &lgr;x has been newly watched.
[0568] More specific details on the above-described operation of the television remote controls 1600 and 1900 can be found in the Computer Program Listing Appendix attached hereto and previously incorporated herein by reference. It is noted that the file “Intuition Intelligence-remote.doc” implements both functionalities for the generalized favorite channel key 1665 and the specialized favorite channel key 1965, with the channel parameter being a switched television channel. It should be noted that the files “Intuition Intelligence-remote.doc” was designed to emulate real-world remote control usage scenarios and to demonstrate the learning and intelligence capability. Only the functions relating to the generalized and specialized favorite channel keys 1665 and 1965 are set forth.
[0569] To this end, the remote control simulation is performed on a personal computer with the Windows 98 OS with Microsoft Access 2000 database support and Media Player. Media Player plays an AVI video file to simulate as if user is watching a program on TV. The Access 2000 database acts as a repository and stores all the lists with all relevant data including the probability values, count of the channel watched, channel number, name, etc., as well as channel number, channel name, channel type, age group, rating, etc. The code and algorithm is implemented in Visual Basic 5.0 with the help of Access 2000 database support.
[0570] As the program has access to more information than a simple remote (which has no Program details, like rating, cast, etc.) it uses a combination of the data available from the cable box or set top box or other mechanisms which can provide the additional information. The program can also implemented without that additional programming information as well. The access to this additional information, however, provides help in demonstrating more sophisticated demo.
[0571] Generalized Multi-User Learning Program (Single Processor Action-Multiple User Actions)
[0572] Hereintobefore, intuitive learning methodologies directed to single-user (i.e., single-teacher) learning scenarios have been described. Referring to FIG. 30, a multi-user learning program 2200 developed in accordance with the present inventions can be generally implemented to provide intuitive learning capability to any variety of processing devices. In this embodiment, multiple users 2205(1)-(3) (here, three) interact with the program 2200 by receiving the same processor action &agr;x from a processor action set &agr; within the program 2200, each independently selecting corresponding user actions &lgr;x1-&lgr;x3 from respective user action sets &lgr;1-&lgr;3 based on the received processor action &agr;x (i.e., user 2205(1) selects a user action &lgr;x1 from the user action set &lgr;1, user 2205(2) selects a user action &lgr;x2 from the user action set &lgr;2, and user 2205(3) selects a user action &lgr;x3 from the user action set &lgr;3), and transmitting the selected user actions &lgr;x1-&lgr;x3 to the program 2200. Again, in alternative embodiments, the users 2205 need not receive the processor action &agr;x to select the respective user actions &lgr;x1-&lgr;x3, the selected user actions &lgr;x1-&lgr;x3 need not be based on the received processor action &agr;x, and/or the processor action &agr;i may be selected in response to the selected user actions &lgr;x1-&lgr;x3. The significance is that processor actions &agr;x and user actions &lgr;x1-&lgr;x3 are selected. The program 2200 is capable of learning based on the measured performance (e.g., success or failure) of the selected processor action &agr;x relative to selected user actions &lgr;x1-&lgr;x3, which, for the purposes of this specification, can be measured as outcome values &bgr;1-&bgr;3. As will be described in further detail below, program 2200 directs its learning capability by dynamically modifying the model that it uses to learn based on a performance index &phgr; to achieve one or more objectives.
[0573] To this end, the program 2200 generally includes a probabilistic learning module 2210 and an intuition module 2215. The probabilistic learning module 2210 includes a probability update module 2220, an action selection module 2225, and an outcome evaluation module 2230. Briefly, the probability update module 2220 uses learning automata theory as its learning mechanism, and is configured to generate and update a game move probability distribution p based on the outcome values &bgr;1-&bgr;3. In this scenario, the probability update module 2220 uses a single stochastic learning automaton with a single input to a multi-teacher environment (with the users 2205(1)-(3) as the teachers), and thus, a single-input, multiple-output (SIMO) model is assumed. Exemplary equations that can be used for the SIMO model will be described in further detail below.
[0574] In essence, the program 2200 collectively learns from the users 2205(1)-(3) notwithstanding that the users 2205(1)-(3) provide independent user actions &lgr;x1-&lgr;x3. The action selection module 2225 is configured to select the processor action &agr;x from the processor action set &agr; based on the probability values contained within the game move probability distribution p internally generated and updated in the probability update module 2220. The outcome evaluation module 2230 is configured to determine and generate the outcome values &bgr;1-&bgr;3 based on the relationship between the selected processor action &agr;x and user actions &lgr;x1-&lgr;x3. The intuition module 2215 modifies the probabilistic learning module 2210 (e.g., selecting or modifying parameters of algorithms used in learning module 2210) based on one or more generated performance indexes &phgr; to achieve one or more objectives. As previously discussed, the performance index &phgr; can be generated directly from the outcome values &bgr;1-&bgr;3 or from something dependent on the outcome values &bgr;1-&bgr;3, e.g., the game move probability distribution p, in which case the performance index &phgr; may be a function of the game move probability distribution p, or the game move probability distribution p may be used as the performance index The modification of the probabilistic learning module 2210 is generally accomplished similarly to that described with respect to the afore-described probabilistic learning module 110. That is, the functionalities of (1) the probability update module 2220 (e.g., by selecting from a plurality of algorithms used by the probability update module 2220, modifying one or more parameters within an algorithm used by the probability update module 2220, transforming or otherwise modifying the game move probability distributions); (2) the action selection module 2225 (e.g., limiting or expanding selection of the action &agr;x corresponding to a subset of probability values contained within the game move probability distributions); and/or (3) the outcome evaluation module 2230 (e.g., modifying the nature of the outcome values &bgr;1-&bgr;3 or otherwise the algorithms used to determine the outcome values &bgr;1-&bgr;3), are modified.
[0575] The various different types of learning methodologies previously described herein can be applied to the probabilistic learning module 2210. The operation of the program 2200 is similar to that of the program 100 described with respect to FIG. 4, with the exception that the program 2200 takes into account all of the selected user actions &lgr;x1-&lgr;x3 when performing the steps. Specifically, referring to FIG. 31, the probability update module 2220 initializes the game move probability distributions (step 2250) similarly to that described with respect to step 150 of FIG. 4. The action selection module 2225 then determines if one or more of the user actions &lgr;x1&lgr;x3 have been selected from the respective user action sets &lgr;1-&lgr;3 (step 2255). If not, the program 2200 does not select a processor action &agr;x from the processor action set &agr; (step 2260), or alternatively selects a processor action &agr;x, e.g., randomly, notwithstanding that none of the user actions &lgr;x1-&lgr;x3 has been selected (step 2265), and then returns to step 2255 where it again determines if one or more of the user actions &lgr;x1-&lgr;x3 have been selected. If one or more of the user actions &lgr;x1-&lgr;x3 have been performed at step 2255, the action selection module 2225 determines the nature of the selected ones of the user actions &lgr;x1-&lgr;x3.
[0576] Specifically, the action selection module 2225 determines whether any of the selected ones of the user actions &lgr;x1-&lgr;x3 are of the type that should be countered with a processor action &agr;x (step 2270). If so, the action selection module 2225 selects a processor action &agr;x from the processor action set &agr; based on the game move probability distribution p (step 2275). After the performance of step 2275 or if the action selection module 2225 determines that none of the selected user actions &lgr;x1-&lgr;x3 is of the type that should be countered with a processor action &agr;x, the action selection module 2225 determines if any of the selected user actions &lgr;x1-&lgr;x3 are of the type that the performance index &phgr; is based on (step 2280).
[0577] If not, the program returns to step 2255 to determine again whether any of the user actions &lgr;x1-&lgr;x3 have been selected. If so, the outcome evaluation module 2230 quantifies the performance of the previously selected processor action &agr;x relative to the currently selected user actions &lgr;x1-&lgr;x3 by generating outcome values &bgr;1-&bgr;3 (step 2285). The intuition module 2215 then updates the performance index &phgr; based on the outcome values &bgr;1-&bgr;3, unless the performance index &phgr; is an instantaneous performance index that is represented by the outcome values &bgr;1-&bgr;3 themselves (step 2290), and modifies the probabilistic learning module 2210 by modifying the functionalities of the probability update module 2220, action selection module 2225, or outcome evaluation module 2230 (step 2295). The probability update module 2220 then, using any of the updating techniques described herein, updates the game move probability distribution p based on the generated outcome values &bgr;1-&bgr;3 (step 2298).
[0578] The program 2200 then returns to step 2255 to determine again whether any of the user actions &lgr;x1-&lgr;x3 have been selected. It should be noted that the order of the steps described in FIG. 31 may vary depending on the specific application of the program 2200.
[0579] Multi-Player Game Program (Single Game Move-Multiple Player Moves)
[0580] Having now generally described the components and functionality of the learning program 2200, we now describe one of its various applications. Referring to FIG. 32, a multiple-player game program 2400 (shown in FIG. 33) developed in accordance with the present inventions is described in the context of a duck hunting game 2300. The game 2300 comprises a computer system 2305, which can be used in an Internet-type scenario. The computer system 2305 includes multiple computers 2310(1)-(3), which merely act as dumb terminals or computer screens for displaying the visual elements of the game 2300 to multiple players 2315(1)-(3), and specifically, a computer animated duck 2320 and guns 2325(1)-(3), which are represented by mouse cursors. It is noted that in this embodiment, the positions and movements of the duck 2320 at any given time are identically displayed on all three of the computer screens 2315(1)-(3). Thus, in essence, each of the players 2315(1)-(3) visualize the same duck 2320 and are all playing against the same duck 2320. As previously noted with respect to the duck 220 and gun 225 of the game 200, the duck 2320 and guns 2325(1)-(3) can be broadly considered to be computer and user-manipulated objects, respectively. The computer system 2305 further comprises a server 2350, which includes memory 2330 for storing the game program 2400, and a CPU 2335 for executing the game program 2400. The server 2350 and computers 2310(1)-(3) remotely communicate with each other over a network 2355, such as the Internet. The computer system 2305 further includes computer mice 2340(1)-(3) with respective mouse buttons 2345(1)-(3), which can be respectively manipulated by the players 2315(1)-(3) to control the operation of the guns 2325(1)-(3).
[0581] It should be noted that although the game 2300 has been illustrated in a multi-computer screen environment, the game 2300 can be embodied in a single-computer screen environment similar to the computer system 205 of the game 200, with the exception that the hardware provides for multiple inputs from the multiple players 2315(1)-(3). The game 2300 can also be embodied in other multiple-input hardware environments, such as a video game console that receives video game cartridges and connects to a television screen, or a video game machine of the type typically found in video arcades.
[0582] Referring specifically to the computer screens 2310(1)-(3), the rules and objective of the duck hunting game 2300 are similar to those of the game 200. That is, the objective of the players 2315(1)-(3) is to shoot the duck 2320 by moving the guns 2325(1)-(3) towards the duck 2320, intersecting the duck 2320 with the guns 2325(1)-(3), and then firing the guns 2325(1)-(3). The objective of the duck 2320, on the other hand, is to avoid from being shot by the guns 2325(1)-(3). To this end, the duck 2320 is surrounded by a gun detection region 2370, the breach of which by any of the guns 2325(1)-(3) prompts the duck 2320 to select and make one of previously described seventeen moves. The game 2300 maintains respective scores 2360(1)-(3) for the players 2315(1)-(3) and scores 2365(1)-(3) for the duck 2320. To this end, if any one of the players 2315(1)-(3) shoots the duck 2320 by clicking the corresponding one of the mouse buttons 2345(1)-(3) while the corresponding one of the guns 2325(1)-(3) coincides with the duck 2320, the corresponding one of the player scores 2360(1)-(3) is increased. In contrast, if any one of the players 2315(1)-(3) fails to shoot the duck 2320 by clicking the corresponding one of the mouse buttons 2345(1)-(3) while the corresponding one of the guns 2325(1)-(3) does not coincide with the duck 2320, the corresponding one of the duck scores 2365(1)-(3) is increased. As previously discussed with respect to the game 200, the increase in the score can be fixed, one of a multitude of discrete values, or a value within a continuous range of values. It should be noted that although the players 2315(1)-(3) have been described as individually playing against the duck 2320, such that the players 2315(1)-(3) have their own individual scores 2360(1)-(3) with corresponding individual duck scores 2365(1)-(3), the game 2300 can be modified, so that the players 2315(1)-(3) can play against the duck 2320 as a team, such that there is only one player score and one duck score that is identically displayed on all three computers 760(1)-(3).
[0583] As will be described in further detail below, the game 2300 increases its skill level by learning the players' 2315(1)-(3) strategy and selecting the duck's 2320 moves based thereon, such that it becomes more difficult to shoot the duck 2320 as the players 2315(1)-(3) become more skillful. The game 2300 seeks to sustain the players' 2315(1)-(3) interest by collectively challenging the players 2315(1)-(3). To this end, the game 2300 continuously and dynamically matches its skill level with that of the players 2315(1)-(3) by selecting the duck's 2320 moves based on objective criteria, such as, e.g., the difference between a function of the player scores 2360(1)-(3) (e.g., the average) and a function (e.g., the average) of the duck scores 2365(1)-(3). In other words, the game 2300 uses this score difference as a performance index &phgr; in measuring its performance in relation to its objective of matching its skill level with that of the game players. Alternatively, the performance index &phgr; can be a function of the game move probability distribution p.
[0584] Referring further to FIG. 33, the game program 2400 generally includes a probabilistic learning module 2410 and an intuition module 2415, which are specifically tailored for the game 2300. The probabilistic learning module 2410 comprises a probability update module 2420, a game move selection module 2425, and an outcome evaluation module 2430. Specifically, the probability update module 2420 is mainly responsible for learning the players' 2315(1)-(3) strategy and formulating a counterstrategy based thereon, with the outcome evaluation module 2430 being responsible for evaluating game moves performed by the game 2300 relative to game moves performed by the players 2315(1)-(3). The game move selection module 2425 is mainly responsible for using the updated counterstrategy to move the duck 2320 in response to moves by the guns 2325(1)-(3). The intuition module 2415 is responsible for directing the learning of the game program 2400 towards the objective, and specifically, dynamically and continuously matching the skill level of the game 2300 with that of the players 2315(1)-(3). In this case, the intuition module 2415 operates on the game move selection module 2425, and specifically selects the methodology that the game move selection module 2425 will use to select a game move &agr;x from the game move set &agr; as will be discussed in further detail below. In the preferred embodiment, the intuition module 2415 can be considered deterministic in that it is purely rule-based. Alternatively, however, the intuition module 2415 can take on a probabilistic nature, and can thus be quasi-deterministic or entirely probabilistic.
[0585] To this end, the game move selection module 2425 is configured to receive player moves &lgr;1x1-&lgr;1x3 from the players 2315(1)-(3), which takes the form of mouse 2340(1)-(3) positions, i.e., the positions of the guns 2325(1)-(3) at any given time. Based on this, the game move selection module 2425 detects whether any one of the guns 2325(1)-(3) is within the detection region 2370, and if so, selects the game move &agr;x from the game move set &agr; and specifically, one of the seventeen moves that the duck 2320 will make.
[0586] Like with the game program 300, the game move selection module 2425 selects the game move &agr;x based on the updated game strategy, and is thus, further configured to receive the game move probability distribution p from the probability update module 2420, and pseudo-randomly selecting the game move &agr;x based thereon. The intuition module 2415 is configured to modify the functionality of the game move selection module 2425 based on the performance index &phgr;, and in this case, the current skill levels of the players 2315(1)-(3) relative to the current skill level of the game 2300. In the preferred embodiment, the performance index &phgr; is quantified in terms of the score difference value &Dgr; between the average of the player scores 2360(1)-(3) and the duck scores 2365(1)-(3). Although in this case the player scores 2360(1)-(3) equally affect the performance index &phgr; in an incremental manner, it should be noted that the effect that these scores have on the performance index &phgr; may be weighted differently. In the manner described above with respect to game 200, the intuition module 2415 is configured to modify the functionality of the game move selection module 2425 by subdividing the game move set &agr; into a plurality of game move subsets &agr;s, selecting one of the game move subsets &agr;s based on the score difference value &Dgr; (or alternatively, based on a series of previous determined outcome values &bgr;1-&bgr;3 or equivalent or some other parameter indicative of the performance index &phgr;). The game move selection module 2425 is configured to pseudo-randomly select a single game move a from the selected game move subset &agr;s.
[0587] The game move selection module 2425 is further configured to receive player moves &lgr;2x1-&lgr;2x3 from the players 2315(1)-(3) in the form of mouse button 2345(1)-(3) click/mouse 2340(1)-(3) position combinations, which indicate the positions of the guns 2325(1)-(3) when they are fired. The outcome evaluation module 2430 is further configured to determine and output outcome values &bgr;1-&bgr;3 that indicate how favorable the selected game move &agr;x in comparison with the received player moves &lgr;2x1-&lgr;2x3 is, respectively.
[0588] As previously described with respect to the game 200, the outcome evaluation module 2430 employs a collision detection technique to determine whether the duck's 2320 last move was successful in avoiding the gunshots, with each of the outcome values &bgr;1-&bgr;3 equaling one of two predetermined values, e.g., “1” if a collision is not detected (i.e., the duck 2320 is not shot), and “0” if a collision is detected (i.e., the duck 2320 is shot), or alternatively, one of a range of finite integers or real numbers, or one of a range of continuous values.
[0589] The probability update module 2420 is configured to receive the outcome values &bgr;1-&bgr;3 from the outcome evaluation module 2430 and output an updated game strategy (represented by game move probability distribution p) that the duck 2320 will use to counteract the players' 2315(1)-(3) strategy in the future. As will be described in further detail below, the game move probability distributionp is updated periodically, e.g., every second, during which each of any number of the players 2315(1)-(3) may provide a corresponding number of player moves &lgr;2x1-&lgr;2x3 In this manner, the player moves &lgr;2x1-&lgr;2x3 asynchronously performed by the players 2315(1)-(3) may be synchronized to a time period. For the purposes of the specification, a player that the probability update module 2420 takes into account when updating the game move probability distribution p at any given time is considered a participating player. It should be noted that in other types of games, where the player moves &lgr;2x need not be synchronized to a time period, such as, e.g., strategy games, the game move probability distribution p may be updated after all players have performed a player move &lgr;2x.
[0590] It is noted that in the preferred embodiment, the intuition module 2415, probability update module 2420, game move selection module 2425, and evaluation module 2430 are all stored in the memory 2330 of the server 2350, in which case, player moves &lgr;1x1-&lgr;1x3, player moves &lgr;2x1-&lgr;2x3, and the selected game moves &agr;x can be transmitted between the user computers 2310(1)-(3) and the server 2350 over the network 2355.
[0591] In this case, the game program 2400 may employ the following unweighted P-type SIMO equations: 33 p j ( k + 1 ) = p j ( k ) - s ( k ) m g j ( p ( k ) ) + ( 1 - s ( k ) m ) h j ( p ( k ) ) , if α ( k ) ≠ α i [ 32 ] p i ( k + 1 ) = p i ( k ) + s ( k ) m ∑ j = 1 j ≠ i n g j ( p ( k ) ) - ( 1 - s ( k ) m ) ∑ j = 1 j ≠ i n h j ( p ( k ) ) , if α ( k ) = α i [ 33 ]
[0592] where p,(k+1), p,(k), gj(p k)), hj(p(k)), i, j, k, and n have been previously defined, s(k) is the number of favorable responses (rewards) obtained from the participating players for game move &agr;x, and m is the number of participating players. It is noted that s(k) can be readily determined from the outcome values &bgr;1-&bgr;3.
[0593] As an example, if there are a total of ten players, seven of which have been determined to be participating, and if two of the participating players shoot the duck 2320 and the other five participating players miss the duck 2320, m will equal 7, and s(k) will equal 5, and thus equations [32] and [33] can be broken down to: 34 p j ( k + 1 ) = p j ( k ) - 5 7 g j ( p ( k ) ) + 2 7 h j ( p ( k ) ) , if α ( k ) ≠ α i [32-1] p i ( k + 1 ) = p i ( k ) + 5 7 ∑ j = 1 j ≠ i n g j ( p ( k ) ) - 2 7 ∑ j = 1 j ≠ i n h j ( p ( k ) ) , if α ( k ) = α i [33-1]
[0594] It should be noted that a single player may perform more than one player move &lgr;2x in a single probability distribution updating time period, and thus be counted as multiple participating players. Thus, if there are three players, more than three participating players may be considered in equation. In any event, the player move sets &lgr;21-&lgr;23 are unweighted in equation [32], and thus each player affects the game move probability distribution p equally.
[0595] If it is desired that each player affects the game move probability distribution p unequally, the player move sets &lgr;21-&lgr;23 can be weighted. For example, player moves &lgr;2x performed by expert players can be weighted higher than player moves &lgr;2x performed by more novice players, so that the more skillful players affect the game move probability distribution p more than the less skillful players. As a result, the relative skill level of the game 2300 will tend to increase even though the skill level of the novice players do not increase. On the contrary, player moves &lgr;2x performed by novice players can be weighted higher than player moves &lgr;2x performed by more expert players, so that the less skillful players affect the game move probability distribution p more than the more skillful players. As a result, the relative skill level of the game 2300 will tend not to increase even though the skill level of the expert players increase.
[0596] In this case, the game program 2400 may employ the following weighted P-type SIMO equations: 35 p j ( k + 1 ) = p j ( k ) - ( ∑ q = 1 m w q I S q ) g j ( p ( k ) ) + ( ∑ q = 1 m w q I F q ) h j ( p ( k ) ) , if α ( k ) ≠ α i [34] p i ( k + 1 ) = p i ( k ) + ( ∑ q = 1 m w q I S q ) ∑ j = 1 j ≠ i n g j ( p ( k ) ) - ( ∑ q = 1 m w q I F q ) ∑ j = 1 j ≠ i n h j ( p ( k ) ) , if α ( k ) = α i [35]
[0597] where px(k+1), px(k), gj(p(k)), hj(p(k)), i, j, k, and n have been previously defined, q is the ordered one of the participating players, m is the number of participating players, wq is the normalized weight of the qth participating player, ISq is a indicator variable that indicates the occurrence of a favorable response associated with the qth participating player, where ISq is 1 to indicate that a favorable response occurred and 0 to indicate that a favorable response did not occur, and IFq is a variable indicating the occurrence of an unfavorable response associated with the qth participating player, where IFq is 1 to indicate that an unfavorable response occurred and 0 to indicate that an unfavorable response did not occur. It is noted that ISq and IFq can be readily determined from the outcome values &bgr;1-&bgr;3.
[0598] As an example, consider Table 14, which sets forth exemplary participation, weighting, and outcome results of ten players given a particular game move &agr;x. 14 TABLE 14 Exemplary Outcome Results for Ten Players in Weighted SIMO Format Weighting Normalized to All Participating Weighting Normalized to Player # Players (q) Participating Players (w) Outcome (S or F) 1 0.05 1 0.077 S 2 0.20 2 0.307 S 3 0.05 — — — 4 0.10 3 0.154 F 5 0.10 — — — 6 0.05 4 0.077 F 7 0.20 — — — 8 0.10 5 0.154 S 9 0.10 6 0.154 S 10 0.05 7 0.077 S
[0599] In this case, 36 ∑ q = 1 m w q I S q = ( .077 ) ( 1 ) + ( .307 ) ( 1 ) + ( .154 ) ( 0 ) + ( 0.77 ) ( 0 ) + ( .154 ) ( 1 ) + ( .154 ) ( 1 ) + ( .077 ) ( 1 ) = .769 ; and &IndentingNewLine; ∑ q = 1 m w q I F q = ( .077 ) ( 0 ) + ( .307 ) ( 0 ) + ( .154 ) ( 1 ) + ( 0.77 ) ( 1 ) + ( .154 ) ( 0 ) + ( .154 ) ( 0 ) + ( .077 ) ( 0 ) = .231 ;
[0600] and thus, equations [34] and [35] can be broken down to:
pj(k+1)=pj(k)−0.769gj(p(k))+0.231hj(p(k)), if &agr;(k)≠&agr;x [34-1] 37 p i ( k + 1 ) = p i ( k ) + 0.769 ∑ j = 1 j ≠ i n g j ( p ( k ) ) - 0.231 ∑ j = 1 j ≠ i n h j ( p ( k ) ) , if α ( k ) = α i [35-1]
[0601] It should be also noted that although the probability update module 2420 may update the game move probability distribution p based on a combination of players participating during a given period of time by employing equations [34]-[35], the probability update module 2420 may alternatively update the game move probability distribution p as each player participates by employing SISO equations [4] and [5]. In general, however, updating the game move probability distribution p on a player-by-player participation basis requires more processing power than updating the game move probability distribution p on a grouped player participation basis. This processing capability becomes more significant as the number of players increases.
[0602] It should also be noted that a single outcome value &bgr; can be generated in response to several player moves &lgr;2. In this case, if less than a predetermined number of collisions are detected, or alternatively, less than a predetermined percentage of collisions are detected based on the number of player moves &lgr;2x received, the outcome evaluation module 2430 will generate an favorable outcome value &Dgr;, e.g., “1”, will be generated. In contrast, if a predetermined number of collisions or more are detected, or alternatively, a predetermined percentage of collisions or more are detected based on the number of player moves &lgr;2x received, the outcome evaluation module 2430 will generate a favorable outcome value &bgr;, e.g., “0.” As will be described in further detail below, a P-type Maximum Probability of Majority Approval (MPMA) SISO equation can be used in this case. Optionally, the extent of the collision or the players that perform the player moves &lgr;2x can be weighted. For example, shots to the head may be weighted higher than shots to the abdomen, or stronger players may be weighted higher than weaker players. Q-type or S-type equations can be used, in which case, the outcome value &bgr; may be a value between “0” and “1”.
[0603] Having now described the structure of the game program 2400, the steps performed by the game program 2400 will be described with reference to FIG. 34. First, the probability update module 2420 initializes the game move probability distribution p and current game move &agr;x (step 2505) similarly to that described in step 405 of FIG. 9. Then, the game move selection module 2425 determines whether any of the player moves &lgr;2x1-&lgr;2x3 have been performed, and specifically whether the guns 2325(1)-(3) have been fired (step 2510). If any of the player moves &lgr;2x1-&lgr;2x3 have been performed, the outcome evaluation module 2430 generates the corresponding outcome values &bgr;1-&bgr;3, as represented by s(k) and m values (unweighted case) or ISq and IFq occurrences (weighted case), for the performed ones of the player moves &lgr;2x1-&lgr;2x3 (step 2515), and the intuition module 2415 then updates the corresponding player scores 2360(1)-(3) and duck scores 2365(1)-(3) based on the corresponding outcome values &bgr;1-&bgr;3 (step 2520), similarly to that described in steps 415 and 420 of FIG. 9. The intuition module 2415 then determines if the given time period to which the player moves &lgr;2x1-&lgr;2x3 are synchronized has expired (step 2521). If the time period has not expired, the game program 2400 will return to step 2510 where the game move selection module 2425 determines again if any of the player moves &lgr;2x1-&lgr;2x3 have been performed. If the time period has expired, the probability update module 2420 then, using the unweighted SIMO equations [32] and [33] or the weighted SIMO equations [34] and [35], updates the game move probability distributions based on the generated outcome values &bgr;1-&bgr;3 (step 2525). Alternatively, rather than synchronize the asynchronous performance of the player moves &lgr;2x1-&lgr;2x3 to the time period at step 921, the probability update module 2420 can update the game move probability distribution p after each of the asynchronous player moves &lgr;2x1-&lgr;2x3 is performed using any of the techniques described with respect to the game program 300. Also, it should be noted that if a single outcome value &bgr; is to be generated for a group of player moves &lgr;2x1-&lgr;2x3, outcome values &bgr;1-&bgr;3 are not generated at step 2520, but rather the single outcome value &bgr; is generated only after the time period has expired at step 2521, and then the game move probability distribution p is updated at step 2525.
[0604] After step 2525, or if none of the player moves &lgr;2x1-&lgr;2x3 has been performed at step 2510, the game move selection module 2425 determines if any of the player moves &lgr;1x1-&lgr;1x3 have been performed, i.e., guns 2325(1)-(3), have breached the gun detection region 270 (step 2530). If none of the guns 2325(1)-(3) has breached the gun detection region 270, the game move selection module 2425 does not select a game move &agr;x from the game move set &agr; and the duck 2320 remains in the same location (step 2535). Alternatively, the game move &agr;x may be randomly selected, allowing the duck 2320 to dynamically wander. The game program 2400 then returns to step 2510 where it is again determined if any of the player moves &lgr;1x1-&lgr;1x3 has been performed. If any of the guns 2325(1)-(3) have breached the gun detection region 270 at step 2530, the intuition module 2415 modifies the functionality of the game move selection module 2425 based on the performance index &phgr;, and the game move selection module 2425 selects a game move &agr;x from the game move a in the manner previously described with respect to steps 440-470 of FIG. 9 (step 2540).
[0605] It should be noted that, rather than use the game move subset selection technique, other afore-described techniques used to dynamically and continuously match the skill level of the players 2315(1)-(3) with the skill level of the game 2300, such as that illustrated in FIG. 10, can be alternatively or optionally be used as well in the game program 2400.
[0606] Generalized Multi-User Learning Program (Multiple Processor Actions-Multiple User Actions)
[0607] Referring to FIG. 35, another multi-user learning program 2600 developed in accordance with the present inventions can be generally implemented to provide intuitive learning capability to any variety of processing devices. In this embodiment, multiple users 2605(1)-(3) (here, three) interact with the program 2600 by respectively receiving processor actions &agr;x1-&agr;x3 from respective processor action subsets &agr;1-&agr;3 within the program 2600, each independently selecting corresponding user actions &lgr;x1-&lgr;x3 from respective user action sets &lgr;1-&lgr;3 based on the received processor actions &agr;x1-&agr;x3 (i.e., user 2605(1) selects a user action &lgr;x1 from the user action set &lgr;1 based on the received processor action &agr;x2, user 2605(2) selects a user action &lgr;x2 from the user action set &lgr;2 based on the received processor action &agr;x2, and user 2605(3) selects a user action &lgr;x3 from the user action set &lgr;3 based on the received processor action &agr;x3), and transmitting the selected user actions &lgr;x1-&lgr;x3 to the program 2600. Again, in alternative embodiments, the users 2605 need not receive the processor actions &agr;x1-&agr;x3, the selected user actions &lgr;x1-&lgr;x3 need not be based on the received processor actions &agr;x1-&agr;x3, and/or the processor actions &agr;x1-&agr;x3 may be selected in response to the selected user actions &lgr;x1-&lgr;x3. The significance is that processor actions &agr;x1-&agr;x3 and user actions &lgr;x1-&lgr;x3 are selected.
[0608] It should be noted that the multi-user learning program 2600 differs from the multi-user learning program 2200 in that the multiple users 2605(1)-(3) can receive multiple processor actions &agr;x1-&agr;x3 from the program 2600 at any given instance, all of which may be different, whereas the multiple users 2205(1)-(3) all receive a single processor action &agr;x from the program 2200. It should also be noted that the number and nature of the processor actions may vary or be the same within the processor action sets &agr;1, &agr;2, and &agr;3 themselves. The program 2600 is capable of learning based on the measured performance (e.g., success or failure) of the selected processor actions &agr;x1-&agr;x3 relative to selected user actions &lgr;x1-&lgr;x3 which, for the purposes of this specification, can be measured as outcome values &bgr;1-&bgr;3. As will be described in further detail below, program 2600 directs its learning capability by dynamically modifying the model that it uses to learn based on performance indexes &phgr;1-&phgr;3 to achieve one or more objectives.
[0609] To this end, the program 2600 generally includes a probabilistic learning module 2610 and an intuition module 2615. The probabilistic learning module 2610 includes a probability update module 2620, an action selection module 2625, and an outcome evaluation module 2630. Briefly, the probability update module 2620 uses learning automata theory as its learning mechanism, and is configured to generate and update an action probability distribution p based on the outcome values &bgr;1-&bgr;3. In this scenario, the probability update module 2620 uses a single stochastic learning automaton with multiple inputs to a multi-teacher environment (with the users 2605(1)-(3) as the teachers), and thus, a multiple-input, multiple-output (MIMO) model is assumed. Exemplary equations that can be used for the MIMO model will be described in further detail below.
[0610] In essence, as with the program 2200, the program 2600 collectively learns from the users 2605(1)-(3) notwithstanding that the users 2605(1)-(3) provide independent user actions user actions &lgr;x1-&lgr;x3. The action selection module 2625 is configured to select the processor actions &agr;x1-&agr;x3 based on the probability values contained within the action probability distribution p internally generated and updated in the probability update module 2620. Alternatively, multiple action selection modules 2625 or multiple portions of the action selection module 2625 may be used to respectively select the processor actions &agr;x1-&agr;x3. The outcome evaluation module 2630 is configured to determine and generate the outcome values &bgr;1-&bgr;3 based on the respective relationship between the selected processor actions &agr;x1-&agr;x3 and user actions &lgr;x1-&lgr;x3. The intuition module 2615 modifies the probabilistic learning module 2610 (e.g., selecting or modifying parameters of algorithms used in learning module 2610) based on the generated performance indexes &phgr;1-&phgr;3 to achieve one or more objectives. Alternatively, a single performance index &phgr; can be used. As previously described, the performance indexes &phgr;1-&phgr;3 can be generated directly from the outcome values &bgr;1-&bgr;3 or from something dependent on the outcome values &bgr;1-&bgr;3, e.g., the action probability distributions, in which case the performance indexes &phgr;1-&phgr;3 may be a function of the action probability distribution p, or the action probability distribution p may be used as the performance indexes &phgr;1-&phgr;3.
[0611] The modification of the probabilistic learning module 2610 is generally accomplished similarly to that described with respect to the afore-described probabilistic learning module 110. That is, the functionalities of (1) the probability update module 2620 (e.g., by selecting from a plurality of algorithms used by the probability update module 2620, modifying one or more parameters within an algorithm used by the probability update module 2620, transforming or otherwise modifying the action probability distribution p); (2) the action selection module 2625 (e.g., limiting or expanding selection of the processor action &agr;x corresponding to a subset of probability values contained within the action probability distribution p); and/or (3) the outcome evaluation module 2630 (e.g., modifying the nature of the outcome values &bgr;1-&bgr;3 or otherwise the algorithms used to determine the outcome values &bgr;1-&bgr;3), are modified.
[0612] The various different types of learning methodologies previously described herein can be applied to the probabilistic learning module 2610. The operation of the program 2600 is similar to that of the program 2200 described with respect to FIG. 31, with the exception that the program 2600 individually responds to the user actions &lgr;x1-&lgr;x3 with processor actions &agr;ix1-&agr;x3 when performing the steps. Specifically, referring to FIG. 36, the probability update module 2620 initializes the action probability distribution p (step 2650) similarly to that described with respect to step 150 of FIG. 4. The action selection module 2625 then determines if one or more of the user actions &lgr;x1-&lgr;x3 have been selected from the user action sets &lgr;1-&lgr;3 (step 2655). If not, the program 2600 does not select processor actions &agr;x1-&agr;x3 from the respective processor action sets &agr;1-&agr;3 (step 2660), or alternatively selects processor actions &agr;x1-&agr;x3, e.g., randomly, notwithstanding that none of the user actions &lgr;x1-&lgr;x3 has been selected (step 2665), and then returns to step 2655 where it again determines if one or more of the user actions &lgr;x1-&lgr;x3 have been selected. If one or more of the user actions &lgr;x1-&lgr;x3 have been selected at step 2655, the action selection module 2625 determines the nature of the selected ones of the user actions &lgr;x1-&lgr;x3.
[0613] Specifically, the action selection module 2625 determines whether any of the selected ones of the user actions &lgr;x1-&lgr;x3 are of the type that should be countered with the corresponding ones of the processor actions &agr;x1-&agr;x3 (step 2670). If so, the action selection module 2625 selects the processor action &agr;x from the corresponding processor action sets &agr;1-&agr;3 based on the action probability distribution p (step 2675). Thus, if user action &lgr;1 was selected and is of the type that should be countered with a processor action &agr;x, a processor action &agr;x1 will be selected from the processor action set &agr;1. If user action &lgr;2 was selected and is of the type that should be countered with a processor action &agr;x, a processor action &agr;x2 will be selected from the processor action set &agr;2. If user action &lgr;3 was selected and is of the type that should be countered with a processor action &agr;x, a processor action &agr;x3 will be selected from the processor action set &agr;3. After the performance of step 2675 or if the action selection module 2625 determines that none of the selected user actions &lgr;x1-&lgr;x3 are of the type that should be countered with a processor action &agr;x, the action selection module 2625 determines if any of the selected user actions &lgr;x1-&lgr;x3 are of the type that the performance indexes &phgr;1-&phgr;3 are based on (step 2680).
[0614] If not, the program 2600 returns to step 2655 to determine again whether any of the user actions &lgr;x1-&lgr;x3 have been selected. If so, the outcome evaluation module 2630 quantifies the performance of the previously corresponding selected processor actions &agr;x1&agr;x3 relative to the currently selected user actions &lgr;x1-&lgr;x3, respectively, by generating outcome values &bgr;1-&bgr;3 (step 2685). The intuition module 2615 then updates the performance indexes &phgr;1-&phgr;3 based on the outcome values &bgr;1-&bgr;3, unless the performance indexes &phgr;1-&phgr;3 are instantaneous performance indexes that are represented by the outcome values &bgr;1-&bgr;3 themselves (step 2690), and modifies the probabilistic learning module 2610 by modifying the functionalities of the probability update module 2620, action selection module 2625, or outcome evaluation module 2630 (step 2695). The probability update module 2620 then, using any of the updating techniques described herein, updates the action probability distribution p based on the generated outcome values &bgr;1-&bgr;3 (step 2698).
[0615] The program 2600 then returns to step 2655 to determine again whether any of the user actions &lgr;x1-&lgr;x3 have been selected. It should be noted that the order of the steps described in FIG. 36 may vary depending on the specific application of the program 2600.
[0616] Multi-Player Game Program (Multiple Game Moves-Multiple Player Moves)
[0617] Having now generally described the components and functionality of the learning program 2600, we now describe one of its various applications. Referring to FIG. 37, a multiple-player game program 2800 developed in accordance with the present inventions is described in the context of a duck hunting game 2700. The game 2700 comprises a computer system 2705, which like the computer system 2305, can be used in an Internet-type scenario, and includes multiple computers 2710(1)-(3), which display the visual elements of the game 2700 to multiple players 2715(1)-(3), and specifically, different computer animated ducks 2720(1)-(3) and guns 2725(1)-(3), which are represented by mouse cursors. It is noted that in this embodiment, the positions and movements of the corresponding ducks 2720(1)-(3) and guns 2725(1)-(3) at any given time are individually displayed on the corresponding computer screens 2715(1)-(3). Thus, in essence, as compared to the game 2300 where each of the players 2315(1)-(3) visualizes the same duck 2320, the players 2715(1)-(3) in this embodiment visualize different ducks 2720(1)-(3) and the corresponding one of the guns 2725(1)-(3). That is, the player 2715(1) visualizes the duck 2720(1) and gun 2725(1), the player 2715(2) visualizes the duck 2720(2) and gun 2725(2), and the player 2715(3) visualizes the duck 2720(3) and gun 2725(3).
[0618] As previously noted with respect to the duck 220 and gun 225 of the game 200, the ducks 2720(1)-(3) and guns 2725(1)-(3) can be broadly considered to be computer and user-manipulated objects, respectively. The computer system 2715 further comprises a server 2750, which includes memory 2730 for storing the game program 2800, and a CPU 2735 for executing the game program 2800. The server 2750 and computers 2710(1)-(3) remotely communicate with each other over a network 2755, such as the Internet. The computer system 2715 further includes computer mice 2740(1)-(3) with respective mouse buttons 2745(1)-(3), which can be respectively manipulated by the players 2715(1)-(3) to control the operation of the guns 2725(1)-(3). As will be described in further detail below, the computers 2710(1)-(3) can be implemented as dumb terminals, or alternatively smart terminals to off-load some of the processing power from the server 2750.
[0619] Referring specifically to the computers 2710(1)-(3), the rules and objective of the duck hunting game 2700 are similar to those of the game 2300. That is, the objective of the players 2715(1)-(3) is to respectively shoot the ducks 2720(1)-(3) by moving the corresponding guns 2725(1)-(3) towards the ducks 2720(1)-(3), intersecting the ducks 2720(1)-(3) with the guns 2725(1)-(3), and then firing the guns 2725(1)-(3). The objective of the ducks 2720(1)-(3), on the other hand, is to avoid from being shot by the guns 2725(1)-(3). To this end, the ducks 2720(1)-(3) are surrounded by respective gun detection regions 2770(1)-(3), the respective breach of which by the guns 2725(1)-(3) prompts the ducks 2720(1)-(3) to select and make one of the previously described seventeen moves. The game 2700 maintains respective scores 2760(1)-(3) for the players 2715(1)-(3) and respective scores 2765(1)-(3) for the ducks 2720(1)-(3). To this end, if the players 2715(1)-(3) respectively shoot the ducks 2720(1)-(3) by clicking the mouse buttons 2745(1)-(3) while the corresponding guns 2725(1)-(3) coincide with the ducks 2720(1)-(3), the player scores 2760(1)-(3) are respectively increased. In contrast, if the players 2715(1)-(3) respectively fail to shoot the ducks 2720(1)-(3) by clicking the mouse buttons 2745(1)-(3) while the guns 2725(1)-(3) do not coincide with the ducks 2720(1)-(3), the duck scores 2765(1)-(3) are respectively increased. As previously discussed with respect to the game 2300, the increase in the scores can be fixed, one of a multitude of discrete values, or a value within a continuous range of values.
[0620] As will be described in further detail below, the game 2700 increases its skill level by learning the players' 2715(1)-(3) strategy and selecting the respective ducks' 2720(1)-(3) moves based thereon, such that it becomes more difficult to shoot the ducks 2720(1)-(3) as the player 2715(1)-(3) becomes more skillful. The game 2700 seeks to sustain the players' 2715(1)-(3) interest by challenging the players 2715(1)-(3). To this end, the game 2700 continuously and dynamically matches its skill level with that of the players 2715(1)-(3) by selecting the ducks' 2720(1)-(3) moves based on objective criteria, such as, e.g., the respective differences between the player scores 2760(1)-(3) and the duck scores 2765(1)-(3). In other words, the game 2700 uses these respective score differences as performance indexes &phgr;1-&phgr;3 in measuring its performance in relation to its objective of matching its skill level with that of the game players.
[0621] Referring further to FIG. 38, the game program 2800 generally includes a probabilistic learning module 2810 and an intuition module 2815, which are specifically tailored for the game 2700. The probabilistic learning module 2810 comprises a probability update module 2820, a game move selection module 2825, and an outcome evaluation module 2830. Specifically, the probability update module 2820 is mainly responsible for learning the players' 2715(1)-(3) strategy and formulating a counterstrategy based thereon, with the outcome evaluation module 2830 being responsible for evaluating game moves performed by the game 2700 relative to game moves performed by the players 2715(1)-(3). The game move selection module 2825 is mainly responsible for using the updated counterstrategy to respectively move the ducks 2720(1)-(3) in response to moves by the guns 2725(1)-(3). The intuition module 2815 is responsible for directing the learning of the game program 2800 towards the objective, and specifically, dynamically and continuously matching the skill level of the game 2700 with that of the players 2715(1)-(3). In this case, the intuition module 2815 operates on the game move selection module 2825, and specifically selects the methodology that the game move selection module 2825 will use to select game moves &agr;x-&agr;x3 from the respective game move sets &agr;1-&agr;3, as will be discussed in further detail below. In the preferred embodiment, the intuition module 2815 can be considered deterministic in that it is purely rule-based. Alternatively, however, the intuition module 2815 can take on a probabilistic nature, and can thus be quasi-deterministic or entirely probabilistic.
[0622] To this end, the game move selection module 2825 is configured to receive player moves &lgr;1x1-&lgr;1x3 from the players 2715(1)-(3), which take the form of mouse 2740(1)-(3) positions, i.e., the positions of the guns 2725(1)-(3) at any given time. Based on this, the game move selection module 2825 detects whether any one of the guns 2725(1)-(3) is within the detection regions 2770(1)-(3), and if so, selects game moves &agr;x1-&agr;x3 from the respective game move sets &agr;1-&agr;3 and specifically, one of the seventeen moves that the ducks 2720(1)-(3) will make.
[0623] The game move selection module 2825 respectively selects the game moves &agr;x1-&agr;x3 based on the updated game strategy, and is thus, further configured to receive the game move probability distributions from the probability update module 2820, and pseudo-randomly selecting the game moves &agr;x1-&agr;x3 based thereon. The intuition module 2815 modifies the functionality of the game move selection module 2825 based on the performance indexes &phgr;1-&phgr;3 and in this case, the current skill levels of the players 2715(1)-(3) relative to the current skill level of the game 2700. In the preferred embodiment, the performance indexes &phgr;1-&phgr;3 are quantified in terms of the respective score difference values &Dgr;1-&Dgr;3 between the player scores 2760(1)-(3) and the duck scores 2765(1)-(3). Although in this case the player scores 2760(1)-(3) equally affect the performance indexes &phgr;1-&phgr;3 in an incremental manner, it should be noted that the effect that these scores have on the performance indexes &phgr;1-&phgr;3 may be weighted differently. In the manner described above with respect to game 200, the intuition module 2815 is configured to modify the functionality of the game move selection module 2825 by subdividing the game move set &agr;1 into a plurality of game move subsets &agr;s1 and selecting one of the game move subsets &agr;s1 based on the score difference value &Dgr;1; subdividing the game move set &agr;2 into a plurality of game move subsets &agr;s2 and selecting one of the game move subsets &agr;s2 based on the score difference value &Dgr;2; and subdividing the game move set &agr;3 into a plurality of game move subsets &agr;s3 and selecting one of the game move subsets &agr;s3 based on the score difference value &Dgr;3 (or alternatively, based on a series of previous determined outcome values &bgr;1-&bgr;3 or some other parameter indicative of the performance indexes &phgr;1-&phgr;3). The game move selection module 2825 is configured to pseudo-randomly select game moves &agr;1l-a13 from the selected ones of the game move subsets &agr;s1-&agr;s3.
[0624] The game move selection module 2825 is further configured to receive player moves &lgr;2x1-&lgr;2x3 from the players 271 5(1)-(3) in the form of mouse button 2745(1)-(3) click/mouse 1040(1)-(3) position combinations, which indicate the positions of the guns 2725(1)-(3) when they are fired. The outcome evaluation module 2830 is further configured to determine and output outcome values &bgr;1-&bgr;3 that indicate how favorable the selected game move &agr;x1, &agr;x2, and &agr;x3 in comparison with the received player moves &lgr;2x1-&lgr;2x3 are, respectively.
[0625] As previously described with respect to the game 200, the outcome evaluation module 2830 employs a collision detection technique to determine whether the ducks' 2720(1)-(3) last moves were successful in avoiding the gunshots, with the outcome values &bgr;1-&bgr;3 equaling one of two predetermined values, e.g., “1” if a collision is not detected (i.e., the ducks 2720(1)-(3) are not shot), and “0” if a collision is detected (i.e., the ducks 1020(1)-(3) are shot), or alternatively, one of a range of finite integers or real numbers, or one of a range of continuous values.
[0626] The probability update module 2820 is configured to receive the outcome values &bgr;1-&bgr;3 from the outcome evaluation module 2830 and output an updated game strategy (represented by game move probability distribution p) that the ducks 2720(1)-(3) will use to counteract the players' 2715(1)-(3) strategy in the future. As will be described in further detail below, the game move probability distribution p is updated periodically, e.g., every second, during which each of any number of the players 2715(1)-(3) may provide one or more player moves &lgr;2x1-&lgr;2x3. In this manner, the player moves &lgr;2x1-&lgr;2x3 asynchronously performed by the players 2715(1)-(3) may be synchronized to a time period. For the purposes of the specification, a player that the probability update module 2820 takes into account when updating the game move probability distribution p at any given time is considered a participating player.
[0627] The game program 2800 may employ the following unweighted P-type MIMO learning methodology: 38 p i ( k + 1 ) = p i ( k ) + s i ( k ) m ∑ j = 1 j ≠ i n g j ( p ( k ) ) - ( r i ( k ) - s i ( k ) ) m ∑ j = 1 j ≠ i n h j ( p ( k ) ) - ∑ j = 1 j ≠ i n s j ( k ) m g i ( p ( k ) ) + ∑ j = 1 j ≠ i n ( r j ( k ) - s j ( k ) ) m h i ( p ( k ) ) [ 36 ]
[0628] where p,(k+1), p,(k), gj(p(k)), hj(p(k)), i,j, k, and n have been previously defined, rx(k) is the total number of favorable (rewards) and unfavorable responses (penalties) obtained from the participating players for game move &agr;x, sx(k) is the number of favorable responses (rewards) obtained from the participating players for game move &agr;x, rj(k) is the total number of favorable (rewards) and unfavorable responses (penalties) obtained from the participating players for game move &agr;j, sj(k) is the number of favorable responses (rewards) obtained from the participating players for game move &agr;j. It is noted that sx(k) can be readily determined from the outcome values &bgr;1-&bgr;3 corresponding to game moves &agr;x and sj(k) can be readily determined from the outcome values &bgr;1-&bgr;3 corresponding to game moves &agr;j.
[0629] As an example, consider Table 15, which sets forth exemplary participation, outcome results of ten players, and game moves &agr;x to which the participating players have responded. 15 TABLE 15 Exemplary Outcome Results for Ten Players in Unweighted MIMO Format Game Move (&agr;1) Player # Responded To Outcome (S or F) 1 &agr;1 S 2 — — 3 &agr;1 F 4 &agr;15 S 5 &agr;2 S 6 — — 7 &agr;2 S 8 &agr;13 F 9 &agr;15 F 10 &agr;2 F
[0630] In this case, m=8, r1(k)=2, s1(k)=1, r2(k)=3, s2(k)=2, r13(k)=1, s13(k)=0, r15(k)=2, s5(k)=1, r3-12, 14, 16-17(k)=0, and r3-12, 14, 16-17(k)=0, and thus, equation [36] can be broken down to:
[0631] for game moves &agr;1, &agr;2, &agr;13, &agr;15: 39 p 1 ( k + 1 ) = p 1 ( k ) + 1 8 ∑ j = 1 j ≠ i n g j ( p ( k ) ) - 1 8 ∑ j = 1 j ≠ i n h j ( p ( k ) ) - 3 8 g 1 ( p ( k ) ) + 3 8 h 1 ( p ( k ) ) p 2 ( k + 1 ) = p 2 ( k ) + 2 8 ∑ j = 1 j ≠ i n g j ( p ( k ) ) - 1 8 ∑ j = 1 j ≠ i n h j ( p ( k ) ) - 2 8 g 2 ( p ( k ) ) + 3 8 h 2 ( p ( k ) ) p 13 ( k + 1 ) = p 13 ( k ) - 1 8 ∑ j = 1 j ≠ i n h j ( p ( k ) ) - 4 8 g 13 ( p ( k ) ) + 3 8 h 13 ( p ( k ) ) p 15 ( k + 1 ) = p 15 ( k ) + 1 8 ∑ j = 1 j ≠ i n g j ( p ( k ) ) - 1 8 ∑ j = 1 j ≠ i n h j ( p ( k ) ) - 3 8 g 15 ( p ( k ) ) + 3 8 h 15 ( p ( k ) )
[0632] for game moves &agr;3-&agr;12, &agr;14, and &agr;16-&agr;17: 40 p i ( k + 1 ) = p i ( k ) - 4 8 g i ( p ( k ) ) + 4 8 h i ( p ( k ) )
[0633] It should be noted that a single player may perform more than one player move &lgr;2x in a single probability distribution updating time period, and thus be counted as multiple participating players. Thus, if there are three players, more than three participating players may be considered in equation. Also, if the game move probability distribution p is only updated periodically over several instances of a player move &lgr;2x, as previously discussed, multiple instances of a player moves &lgr;2x will be counted as multiple participating players. Thus, if three player moves &lgr;2x from a single player are accumulated over a period of time, these player moves &lgr;2x will be treated as if three players had each performed a single player move &lgr;22x.
[0634] In any event, the player move sets &lgr;21-&lgr;23 are unweighted in equation [36], and thus each player affects the game move probability distribution p equally. As with the game program 2400, if it is desired that each player affects the game move probability distribution p unequally, the player move sets &lgr;21-&lgr;23 can be weighted. In this case, the game program 2800 may employ the following weighted P-type MIMO learning methodology: 41 p i ( k + 1 ) = p i ( k ) + ( ∑ q = 1 m w q I Si q ) ( ∑ j = 1 j ≠ i n g j ( p ( k ) ) ) - ( ∑ q = 1 m w q I Fi q ) ( ∑ j = 1 j ≠ i n h j ( p ( k ) ) ) - ( ∑ q = 1 m ∑ j = 1 j ≠ i n w q I Sj q g i ( p ( k ) ) ) + ( ∑ q = 1 m ∑ j = 1 j ≠ i n w q I Fj q h i ( p ( k ) ) ) [ 37 ]
[0635] where p,(k+1), px(k), gj(p(k)), hj(p(k)), i, j, k, and n have been previously defined, q is the ordered one of the participating players, m is the number of participating players, wq is the normalized weight of the qth participating player, Isixq is a variable indicating the occurrence of a favorable response associated with the qth participating player and game move &agr;x, and ISjq is a variable indicating the occurrence of a favorable response associated with the qth participating player and game move &agr;j, IFixq is a variable indicating
[0636] the occurrence of an unfavorable response associated with the qth participating player and game move &agr;x, and IFjxq is a variable indicating the occurrence of an unfavorable response associated with the qth participating player and game move &agr;j. It is noted that ISq and IFq can be readily determined from the outcome values &bgr;1-&bgr;3.
[0637] As an example, consider Table 16, which sets forth exemplary participation, outcome results of ten players, weighting of players, and game moves &agr;x to which the participating players have responded. 16 TABLE 16 Exemplary Outcome Results for Ten Players in Weighted MIMO Format Weighting Normalized to All Participating Game Move(&agr;1) Weighting Normalized to Outcome Player # Players (q) Responded To Participating Players (w) (S or F) 1 0.05 1 &agr;1 0.067 S 2 0.20 — — — — 3 0.05 2 &agr;1 0.067 F 4 0.10 3 &agr;15 0.133 S 5 0.10 4 &agr;2 0.133 S 6 0.05 — — — — 7 0.20 5 &agr;2 0.267 S 8 0.10 6 &agr;13 0.133 F 9 0.10 7 &agr;15 0.133 F 10 0.05 8 &agr;2 0.067 F
[0638] In this case, 42 ∑ q = 1 m w q I S 1 q = w 1 I S1 1 = ( .067 ) ( 1 ) = 0.067 ; ∑ q = 1 m w q I S 2 q = w 5 I S2 5 + w 7 I S2 7 = ( .133 ) ( 1 ) + ( 0.267 ) ( 1 ) = 0.400 ; ∑ q = 1 m w q I S 13 q = 0 ; ∑ q = 1 m w q I S 15 q = w 4 I S15 4 = ( .133 ) ( 1 ) = 0.133 ; ∑ q = 1 m w q I F 1 q = w 3 I F1 3 = ( .067 ) ( 1 ) = 0.067 ; ∑ q = 1 m w q I F 2 q = w 10 I F2 10 = ( .067 ) ( 1 ) = 0.067 ; ∑ q = 1 m w q I F 13 q = w 8 I F13 8 = ( .133 ) ( 1 ) = 0.133 ∑ q = 1 m w q I F 15 q = w 9 I F15 9 = ( .133 ) ( 1 ) = 0.133 ;
[0639] and thus, equation [37] can be broken down to: for game moves &agr;1, &agr;2, &agr;13, &agr;15: 43 p 1 ( k + 1 ) = p 1 ( k ) + 0.067 ∑ j = 1 j ≠ i n g j ( p ( k ) ) - 0.067 ∑ j = 1 j ≠ i n h j ( p ( k ) ) - 0.533 g 1 ( p ( k ) ) + 0.333 h 1 ( p ( k ) ) p 2 ( k + 1 ) = p 2 ( k ) + 0.400 ∑ j = 1 j ≠ i n g j ( p ( k ) ) - 0.067 ∑ j = 1 j ≠ i n h j ( p ( k ) ) - 0.200 g 2 ( p ( k ) ) + 0.333 h 2 ( p ( k ) ) p 13 ( k + 1 ) = p 13 ( k ) - 0.133 ∑ j = 1 j ≠ i n h j ( p ( k ) ) - 0.600 g 13 ( p ( k ) ) + 0.267 h 13 ( p ( k ) ) p 15 ( k + 1 ) = p 15 ( k ) + 0.133 ∑ j = 1 j ≠ i n g j ( p ( k ) ) - 0.133 ∑ j = 1 j ≠ i n h j ( p ( k ) ) - 0.467 g 15 ( p ( k ) ) + 0.267 h 15 ( p ( k ) )
[0640] for game moves &agr;3-&agr;12, &agr;14, and &agr;16-&agr;17:
px(k+1)=px(k)−0.600gx(p(k))+0.400hx(p(k))
[0641] It should be noted that the number of players and game moves &agr;x may be dynamically altered in the game program 2800. For example, the game program 2800 may eliminate weak players by learning the weakest moves of a player and reducing the game score for that player. Once a particular metric is satisfied, such as, e.g., the game score for the player reaches zero or the player loses five times in row, that player is eliminated. As another example, the game program 2800 may learn each players' weakest and strongest moves, and then add a game move &agr;x for the corresponding duck if the player executes a weak move, and eliminate a game move &agr;x for the corresponding duck if the player executes a strong move. In effect, the number of variables within the learning automaton can be increased or decreased. For this we can employ the pruning/growing (expanding) learning algorithms.
[0642] Having now described the structure of the game program 2800, the steps performed by the game program 2800 will be described with reference to FIG. 39. First, the probability update module 2820 initializes the game move probability distribution p and current player moves &lgr;2x1-&lgr;2x3, (step 2905) similarly to that described in step 405 of FIG. 9. Then, the game move selection module 2825 determines whether any of the player moves &lgr;2x1-&lgr;2x3 have been performed, and specifically whether the guns 2725(1)-(3) have been fired (step 2910). If any of the &lgr;2x1, &lgr;2x2, and &lgr;2x3 have been performed, the outcome evaluation module 2830 generates the corresponding outcome values &bgr;1-&bgr;3, as represented by s(k), r(k) and m values (unweighted case) or ISq and IFq occurrences (weighted case), for the performed ones of the player moves &lgr;2x1-&lgr;2x3 and corresponding game moves &agr;x1-&agr;x3 (step 2915), and the intuition module 2815 then updates the corresponding player scores 2760(1)-(3) and duck scores 2765(1)-(3) based on the outcome values &bgr;1-&bgr;3 (step 2920), similarly to that described in steps 415 and 420 of FIG. 9. The intuition module 2815 then determines if the given time period to which the player moves &lgr;2x1-&lgr;2x3 are synchronized has expired (step 2921). If the time period has not expired, the game program 2800 will return to step 2910 where the game move selection module 2825 determines again if any of the player moves &lgr;2x1-&lgr;2x3 have been performed. If the time period has expired, the probability update module 2820 then, using the unweighted MIMO equation [36] or the weighted MIMO equation [37], updates the game move probability distribution p based on the outcome values &bgr;1-&bgr;3 (step 2925). Alternatively, rather than synchronize the asynchronous performance of the player moves &lgr;2x1-&lgr;2x3 to the time period at step 2921, the probability update module 2820 can update the game move probability distribution p after each of the asynchronous player moves &lgr;2x1-&lgr;2x3 is performed using any of the techniques described with respect to the game program 300.
[0643] After step 2925, or if none of the player moves &lgr;2x1-&lgr;2x3 has been performed at step 2910, the game move selection module 2825 determines if any of the player moves &lgr;1x1-&lgr;1x3 have been performed, i.e., guns 2725(1)-(3), have breached the gun detection regions 2770(1)-(3) (step 2930). If none of the guns 2725(1)-(3) have breached the gun detection regions 2770(1)-(3), the game move selection module 2825 does not select any of the game moves &agr;x1-&agr;x3 from the respective game move sets &agr;1-&agr;3, and the ducks 2720(1)-(3) remain in the same location (step 2935). Alternatively, the game moves &agr;x-&agr;x3 may be randomly selected, respectively allowing the ducks 2720(1)-(3) to dynamically wander. The game program 2800 then returns to step 2910 where it is again determined if any of the player moves &lgr;1x1-&lgr;1x3 have been performed. If any of the guns 2725(1)-(3) have breached the gun detection regions 2770(1)-(3) at step 2930, the intuition module 2815 modifies the functionality of the game move selection module 2825, and the game move selection module 2825 selects the game moves &agr;x1-&agr;x3 from the game move sets &agr;1-&agr;3 that correspond to the breaching guns 2725(1)-(3) based on the corresponding performance indexes &phgr;1-&phgr;3 in the manner previously described with respect to steps 440-470 of FIG. 9 (step 2940).
[0644] It should be noted that, rather than use the game move subset selection technique, other afore-described techniques used to dynamically and continuously match the skill level of the players 2715(1)-(3) with the skill level of the game 2700, such as that illustrated in FIG. 10, can be alternatively or optionally be used as well in the game.
[0645] Referring back to FIG. 37, it is noted that the network 2755 is used to transmit information between the user computers 2710(1)-(3) and the server 2750. The nature of this information will depend on how the various modules are distributed amongst the user computers 2710(1)-(3) and the server 2750. In the preferred embodiment, the intuition module 2815 and probability update module 2820 are located within the memory 2730 of the server 2750. Depending on the processing capability of the CPU 2735 of the server 2750 and the anticipated number of players, the game move selection module 2825 and/or game evaluation module 2830 can be located within the memory 2730 of the server 2750 or within the computers 2710(1)-(3).
[0646] For example, if the CPU 2735 has a relatively quick processing capability and the anticipated number of players is low, all modules can be located within the server 2750. In this case, and with reference to FIG. 40, all processing, such as, e.g., selecting game moves &agr;x1-&agr;x3, generating outcome values &bgr;1-&bgr;3, and updating the game move probability distribution p, will be performed in the server 2750. Over the network 2755, selected game moves &agr;x1-&agr;x3 will be transmitted from the server 2750 to the respective user computers 2710(1)-(3), and performed player moves &lgr;1x1-&lgr;1x3 and game moves &lgr;2x1-&lgr;2x3 will be transmitted from the respective user computers 2710(1)-(3) to the server 2750.
[0647] Referring now to FIG. 41, if it is desired to off-load some of the processing functions from the server 2750 to the computers 2710(1)-(3), the game move selection modules 2825 can be stored in the computers 2710(1)-(3), in which case, game move subsets &agr;x1-&agr;x3 can be selected by the server 2750 and then transmitted to the respective user computers 2710(1)-(3) over the network 2755. The game moves &agr;x1-&agr;x3 can then be selected from the game move subsets &agr;s1-&agr;s3 by the respective computers 2710(1)-(3) and transmitted to the server 2750 over the network 2755. In this case, performed player moves &lgr;1x1-&lgr;1x3 need not be transmitted from the user computers 2710(1)-(3) to the server 2750 over the network 2755, since the game moves &agr;x1-&agr;x3 are selected within the user computers 2710(1)-(3).
[0648] Referring to FIG. 42, alternatively or in addition to game move selection modules 2825, outcome evaluation modules 2830 can be stored in the user computers 2710(1)-(3), in which case, outcome values &bgr;1-&bgr;3 can be generated in the respective user computers 2710(1)-(3) and then be transmitted to the server 2750 over the network 2755. It is noted that in this case, performed player moves &lgr;2x1-&lgr;2x3 need not be transmitted from the user computers 2710(1)-(3) to the server 2750 over the network 2755.
[0649] Referring now to FIG. 43, if it is desired to off-load even more processing functions from the server 2750 to the computers 2710(1)-(3), portions of the intuition module 2815 may be stored in the respective computers 2710(1)-(3). In this case, the probability distribution p can be transmitted from the server 2750 to the respective computers 2710(1)-(3) over the network 2755. The respective computers 2710(1)-(3) can then select game move subsets &agr;s1-&agr;s3, and select game moves &agr;x1-&agr;x3 from the selected game move subsets &agr;s1-&agr;s3. If the outcome evaluation module 2830 is stored in the server 2750, the respective computers 2710(1)-(3) will then transmit the selected game moves &agr;x1-&agr;x3 to the server 2750 over the network 2755. If outcome evaluation modules 2830 are stored in the respective user computers 2710(1)-(3), however, the computers 2710(1)-(3) will instead transmit outcome values &bgr;1-&bgr;3 to the server 2750 over the network 2755.
[0650] To even further reduce the processing needs for the server 2750, information is not exchanged over the network 2755 in response to each performance of player moves &lgr;2x1-&lgr;2x3, but rather only after a number of player moves &lgr;2x1-&lgr;2x3 has been performed. For example, if all processing is performed in the server 2750, the performed player moves &lgr;2x1-&lgr;2x3 can be accumulated in the respective user computers 2710(1)-(3) and then transmitted to the server 2750 over the network 2755 only after several player moves &lgr;2x1-&lgr;2x3 have been performed. If the game move selection modules 2825 are located in the respective user computers 2710(1)-(3), both performed player moves &lgr;2x1-&lgr;2x3 and selected game moves &agr;x1-&agr;x3 can be accumulated in the user computers 2710(1)-(3) and then transmitted to the server 2750 over the network 2755. If the outcome evaluation modules 2830 are located in respective user computers 2710(1)-(3), outcome values &bgr;1-&bgr;3 can be accumulated in the user computers 2710(1)-(3) and then transmitted to the server 2750 over the network 2755. In all of these cases, the server 2750 need only update the game move probability distribution p periodically, thereby reducing the processing of the server 2750.
[0651] Like the previously described probability update module 2420, the probability update module 2820 may alternatively update the game move probability distributions as each player participates by employing SISO equations [4] and [5]. In the scenario, the SISO equations [4] and [5] will typically be implemented in a single device that serves the players 2715(1)-(3), such as the server 2750. Alternatively, to reduce the processing requirements in the server 2750, the SISO equations [4] and [5] can be implemented in devices that are controlled by the players 2715(1)-(3), such as the user computers 2710(1)-(3).
[0652] In this case, and with reference to FIG. 44, separate probability distributions p1-p3 are generated and updated in the respective user computers 2710(1)-(3) using SISO equations. Thus, all of the basic functionality, such as performing player moves &lgr;1x1-&lgr;1x3 and &lgr;2x1-&lgr;2x3, subdividing and selecting game move subsets &agr;s1-&agr;s3 and &agr;x1-&agr;x3, and updating the game move probability distributions p1-p3, are performed in the user computers 2710(1)-(3). For each of the user computers 2710(1)-(3), this process can be the same as those described above with respect to FIGS. 9 and 10. The server 2750 is used to maintain some commonality amongst different game move probability distributions p1-p3 being updated in the respective user computers 2710(1)-(3). This may be useful, e.g., if the players 2715(1)-(3) are competing against each other and do not wish to be entirely handicapped by exhibiting a relatively high level of skill. Thus, after several iterative updates, the respective user computers 2710(1)-(3) can periodically transmit their updated probability distributions p1-p3 to the server 2750 over the network 2755. The server 2750 can then update a centralized probability distribution p based on the recently received probability distributions p1-p3, and preferably a weighted average of the probability distributions p1-p3. The weights of the game move probability distributions p1-p3 may depend on, e.g., the number of times the respective game move probability distributions p1-p3 have been updated at the user computers 2710(1)-(3).
[0653] Thus, as the number of player moves &lgr;2x performed at a particular user computer 2710 increases relative to other user computers 2710, the effect that the iteratively updated game move probability distribution p transmitted from this user computer 2710 to the server 2750 has on central game move probability distribution pc will correspondingly increase. Upon generating the centralized probability distribution pc, the server 2750 can then transmit it to the respective user computers 2710(1)-(3). The user computers 2710(1)-(3) can then use the centralized probability distribution p as their initial game move probability distributions p1-p3, which are then iteratively updated. This process will then repeated.
[0654] Generalized Multi-User Learning Program With Multiple Probability Distributions
[0655] Referring to FIG. 45, another multi-user learning program 3000 developed in accordance with the present inventions can be generally implemented to provide intuitive learning capability to any variety of processing devices. Multiple sets of users 3005(1)-(2), 1405(3)-(4), and 1405(5)-(6) (here three sets of two users each) interact with the program 3000 by respectively receiving processor actions &agr;x1-&agr;x6 from respective processor action sets &agr;1-&agr;6 within the program 3000, selecting user actions &lgr;x1-&lgr;x6 from the respective user action sets &lgr;1-&lgr;6 based on the received processor actions &agr;x1-&agr;x6, and transmitting the selected user actions &lgr;x1-&lgr;x6 to the program 3000. Again, in alternative embodiments, the users 3005 need not receive the processor actions &agr;x1-&agr;x6, the selected user actions &lgr;x1-&lgr;x6 need not be based on the received processor actions &agr;x1-&agr;x6, and/or the processor actions &agr;x1-&agr;x6 may be selected in response to the selected user actions &lgr;x1-&lgr;x6. The significance is that processor actions &agr;x1-&agr;x6 and user actions &lgr;x1-&lgr;x6 are selected.
[0656] The program 3000 is capable of learning based on the measured performance (e.g., success or failure) of the selected processor actions &agr;x1-&agr;x6 relative to selected user actions &lgr;x1-&lgr;x6, which, for the purposes of this specification, can be measured as outcome values &bgr;1-&bgr;6. As will be described in further detail below, program 3000 directs its learning capability by dynamically modifying the model that it uses to learn based on performance indexes &phgr;1-&phgr;6 to achieve one or more objectives.
[0657] To this end, the program 3000 generally includes a probabilistic learning module 3010 and an intuition module 3015. The probabilistic learning module 3010 includes a probability update module 3020, an action selection module 3025, and an outcome evaluation module 3030. The program 3000 differs from the program 2600 in that the probability update module 3020 is configured to generate and update multiple action probability distributions p1-p3 (as opposed to a single probability distribution p) based on respective outcome values &bgr;1-&bgr;2, &bgr;3-&bgr;4, and &bgr;5-&bgr;6. In this scenario, the probability update module 3020 uses multiple stochastic learning automatons, each with multiple inputs to a multi-teacher environment (with the users 3005(1)-(6) as the teachers), and thus, a MIMO model is assumed for each learning automaton. Thus, users 3005(1)-(2), users 3005(3)-(4), and users 3005(5)-(6) are respectively associated with action probability distributions p1-p3, and therefore, the program 3000 can independently learn for each of the sets of users 3005(1)-(2), users 3005(3)-(4), and users 3005(5)-(6). It is noted that although the program 3000 is illustrated and described as having a multiple users and multiple inputs for each learning automaton, multiple users with single inputs to the users can be associated with each learning automaton, in which case a SIMO model is assumed for each learning automaton, or a single user with a single input to the user can be associated with each learning automaton, in which case a SISO model can be associated for each learning automaton.
[0658] The action selection module 3025 is configured to select the processor actions &agr;x1-&agr;i2, &agr;x3-&agr;x4, and &agr;x5-&agr;x6 from respective action sets &agr;1-&agr;2, &agr;3-&agr;4, and &agr;5-&agr;6 based on the probability values contained within the respective action probability distributions p1-p3 internally generated and updated in the probability update module 3020. The outcome evaluation module 3030 is configured to determine and generate the outcome values &bgr;1-&bgr;6 based on the respective relationship between the selected processor actions &agr;x1-&agr;x6 and user actions &lgr;x1-&lgr;x6. The intuition module 3015 modifies the probabilistic learning module 3010 (e.g., selecting or modifying parameters of algorithms used in learning module 3010) based on the generated performance indexes &phgr;1-&phgr;6 to achieve one or more objectives. As previously described, the performance indexes &phgr;1-&phgr;6 can be generated directly from the outcome values &bgr;1-&bgr;6 or from something dependent on the outcome values &bgr;1-&bgr;6 e.g., the action probability distributions p1-p3, in which case the performance indexes &phgr;1-&phgr;2, &phgr;3-&phgr;4, and &phgr;5-&phgr;6 maybe a function of the action probability distributions p1-p3, or the action probability distributions p1-p3 may be used as the performance indexes &phgr;1-&phgr;2, &phgr;3-&phgr;4, and &phgr;5-&phgr;6. The modification of the probabilistic learning module 3010 is generally accomplished similarly to that described with respect to the afore-described probabilistic learning module 110. That is, the functionalities of (1) the probability update module 3020 (e.g., by selecting from a plurality of algorithms used by the probability update module 3020, modifying one or more parameters within an algorithm used by the probability update module 3020, transforming or otherwise modifying the action probability distributions p1-p3); (2) the action selection module 3025 (e.g., limiting or expanding selection of the processor actions &agr;x 1-&agr;x2, &agr;x3-&agr;x4, and &agr;x5-&agr;x6 corresponding to subsets of probability values contained within the action probability distributions p1-p3); and/or (3) the outcome evaluation module 3030 (e.g., modifying the nature of the outcome values &bgr;1-&bgr;6 or otherwise the algorithms used to determine the outcome values &bgr;1-&bgr;6), are modified.
[0659] The various different types of learning methodologies previously described herein can be applied to the probabilistic learning module 3010. The steps performed by the program 3000 are similar to that described with respect to FIG. 36, with the exception that the game program 3000 will independently perform the steps of the flow diagram for each of the sets of users 3005(1)-(2), 1405(3)-(4), and 1405(5)-(6). For example, the program 3000 will execute one pass through the flow for users 3005(1)-(2) (and thus the first probability distribution p1), then one pass through the flow for users 3005(3)-(4) (and thus the first probability distribution p2) and then one pass through the flow for users 3005(5)-(6) (and thus the first probability distribution p3).
[0660] Alternatively, the program 3000 can combine the steps of the flow diagram for the users 3005(1)-(6). For example, referring to FIG. 46, the probability update module 3020 initializes the action probability distributions p1-p3 (step 3050) similarly to that described with respect to step 150 of FIG. 4. The action selection module 3025 then determines if one or more of the user actions &lgr;x1-&lgr;x6 have been selected from the respective user action sets &lgr;1-&lgr;6 (step 3055). If not, the program 3000 does not select the processor actions &agr;x1-&agr;x6 from the processor action sets &agr;1-&agr;6 (step 3060), or alternatively selects processor actions &agr;x 1-&agr;x6, e.g., randomly, notwithstanding that none of the user actions &lgr;x1-&lgr;x6 have been selected (step 3065), and then returns to step 3055 where it again determines if one or more of the user actions &lgr;x1-&lgr;x6 have been selected. If one or more of the user actions &lgr;x1-&lgr;x6 have been selected at step 3055, the action selection module 3025 determines the nature of the selected ones of the user actions &lgr;x1-&lgr;x6.
[0661] Specifically, the action selection module 3025 determines whether any of the selected ones of the user actions &lgr;x1-&lgr;x6 are of the type that should be countered with the corresponding ones of the processor actions &agr;x1-&agr;x6 (step 3070). If so, the action selection module 3025 selects processor actions &agr;x from the corresponding processor action sets &agr;1-&agr;2, &agr;3-&agr;4, and &agr;5-&agr;6 based on the corresponding one of the action probability distributions p1-p3 (step 3075). Thus, if either of the user actions &lgr;x1 and &lgr;x2 is selected and is of the type that should be countered with a processor action &agr;x, processor actions &agr;x1 and &agr;x2 will be selected from the corresponding processor action sets &agr;1 and &agr;2 based on the probability distribution p1. If either of the user actions &lgr;x3 and &lgr;x4 is selected and is of the type that should be countered with a processor action &agr;i, processor actions &agr;x3 and &agr;x4 will be selected from the corresponding processor action sets &agr;3 and &agr;4 based on the probability distribution p2. If either of the user actions &lgr;x5 and &lgr;x6 is selected and is of the type that should be countered with a processor action &agr;x, processor actions &agr;x5 and &agr;x6 will be selected from the corresponding processor action sets &agr;5 and &agr;6 based on the probability distribution p3. After the performance of step 3075 or if the action selection module 3025 determines that none of the selected ones of the user actions &lgr;x1-&lgr;x6 is of the type that should be countered with a processor action &agr;x, the action selection module 3025 determines if any of the selected ones of the user actions &lgr;x1-&lgr;x6 are of the type that the performance indexes &phgr;1-&phgr;6 are based on (step 3080).
[0662] If not, the program 3000 returns to step 3055 to determine again whether any of the user actions &lgr;x1-&lgr;x6 have been selected. If so, the outcome evaluation module 3030 quantifies the performance of the previously corresponding selected processor actions &agr;x1-&agr;x6 relative to the selected ones of the current user actions &lgr;x1-&lgr;x6, respectively, by generating outcome values &bgr;1-&bgr;6 (step 3085). The intuition module 3015 then updates the performance indexes &phgr;1-&phgr;6 based on the outcome values &bgr;1-&bgr;6, unless the performance indexes &phgr;1-&phgr;6 are instantaneous performance indexes that are represented by the outcome values &bgr;1-&bgr;6 themselves (step 3090), and modifies the probabilistic learning module 3010 by modifying the functionalities of the probability update module 3020, action selection module 3025, or outcome evaluation module 3030 (step 3095). The probability update module 3020 then, using any of the updating techniques described herein, updates the respective action probability distributions p1-p3 based on the generated outcome values &bgr;1-&bgr;2, &bgr;3-&bgr;4, and &bgr;5-&bgr;6 (step 3098).
[0663] The program 3000 then returns to step 3055 to determine again whether any of the user actions &lgr;x1-&lgr;x6 have been selected. It should also be noted that the order of the steps described in FIG. 46 may vary depending on the specific application of the program 3000.
[0664] Multi-Player Game Program with Multiple Probability Distributions
[0665] Having now generally described the components and functionality of the learning program 3000, we now describe one of its various applications. Referring to FIG. 47, a multiple-player game program 3200 developed in accordance with the present inventions is described in the context of a duck hunting game 3100. The game 3100 is similar to the previously described game 2700 with the exception that three sets of players (players 3115(1)-(2), 3115(3)-(4), and 3115(5)-(6)) are shown interacting with a computer system 3105, which like the computer systems 2305 and 2705, can be used in an Internet-type scenario. Thus, the computer system 3105 includes multiple computers 3110(1)-(6), which display computer animated ducks 3120(1)-(6) and guns 3125(1)-(6). The computer system 3105 further comprises a server 3150, which includes memory 3130 for storing the game program 3200, and a CPU 1535 for executing the game program 3200. The server 3150 and computers 3110(1)-(6) remotely communicate with each other over a network 3155, such as the Internet. The computer system 3105 further includes computer mice 3140(1)-(6) with respective mouse buttons 3145(1)-(6), which can be respectively manipulated by the players 3115(1)-(6) to control the operation of the guns 3125(1)-(6). The ducks 3120(1)-(6) are surrounded by respective gun detection regions 3170(1)-(6). The game 3100 maintains respective scores 3160(1)-(6) for the players 3115(1)-(6) and respective scores 3165(1)-(6) for the ducks 3120(1)-(6).
[0666] As will be described in further detail below, the players 3115(1)-(6) are divided into three sets based on their skill levels (e.g., novice, average, and expert). The game 3100 treats the different sets of players 3115(1)-(6) differently in that it is capable of playing at different skill levels to match the respective skill levels of the players 3115(1)-(6). For example, if players 3115(1)-(2) exhibit novice skill levels, the game 3100 will naturally play at a novice skill level for players 3115(1)-(2). If players 3115(3)-(4) exhibit average skill levels, the game 3100 will naturally play at an average skill level for players 311 5(3)-(4). If players 3115(5)-(6) exhibit expert skill levels, the game 3100 will naturally play at an expert skill level for players 3115(5)-(6). The skill level of each of the players 3115(1)-(6) can be communicated to the game 3100 by, e.g., having each player manually input his or her skill level prior to initiating play with the game 3100, and placing the player into the appropriate player set based on the manual input, or sensing each player's skill level during game play and dynamically placing that player into the appropriate player set based on the sensed skill level. In this manner, the game 3100 is better able to customize itself to each player, thereby sustaining the interest of the players 3115(1)-(6) notwithstanding the disparity of skill levels amongst them.
[0667] Referring further to FIG. 48, the game program 3200 generally includes a probabilistic learning module 3210 and an intuition module 3215, which are specifically tailored for the game 3100. The probabilistic learning module 3210 comprises a probability update module 3220, a game move selection module 3225, and an outcome evaluation module 3230. The probabilistic learning module 3210 and intuition module 3215 are configured in a manner similar to the learning module 2810 and intuition module 2815 of the game program 2800. To this end, the game move selection module 3225 is configured to receive player moves &lgr;1x1-&lgr;1x6 from the players 3115(1)-(6), which take the form of mouse 3140(1)-(6) positions, i.e., the positions of the guns 3125(1)-(6) at any given time. Based on this, the game move selection module 3225 detects whether any one of the guns 3125(1)-(6) is within the detection regions 3170(1)-(6), and if so, selects game moves &agr;x1-&agr;x6 from the respective game move sets &agr;1-&agr;6 and specifically, one of the seventeen moves that the ducks 3120(1)-(6) will make. The game move selection module 3225 respectively selects the game moves &agr;x1-&agr;x2, &agr;x3-&agr;x4, and &agr;x5-&agr;x6 based on game move probability distributions p1-p3 received from the probability update module 3220.
[0668] Like the intuition module 2815, the intuition module 3215 modifies the functionality of the game move selection module 3225 by subdividing the game move set &agr;1-&agr;6 into pluralities of game move subsets &agr;s1-&agr;s6 and selecting one of each of the pluralities of game move subsets &agr;s1-&agr;s6 based on the respective score difference values &Dgr;1-&Dgr;6. The game move selection module 3225 is configured to pseudo-randomly select game moves &agr;x1-&agr;x6 from the selected ones of the game move subsets &agr;s1-&agr;s6.
[0669] The game move selection module 3225 is further configured to receive player moves &lgr;2x1-&lgr;2x6 from the players 3115(1)-(6) in the form of mouse button 1545(1)-(6) click/mouse 3140(1)-(6) position combinations, which indicate the positions of the guns 3125(1)-(6) when they are fired. The outcome evaluation module 3230 is further configured to determine and output outcome values &bgr;1-&bgr;6 that indicate how favorable the selected game moves &agr;x1-&agr;x6 in comparison with the received player moves &lgr;2x1-&lgr;2x6, respectively.
[0670] The probability update module 3220 is configured to receive the outcome values &bgr;1-&bgr;6 from the outcome evaluation module 3230 and output an updated game strategy (represented by game move probability distributions p1-p3) that the ducks 3120(1)-(6) will use to counteract the players' 1515(1)-(6) strategy in the future. Like the game move probability distributionp updated by the probability update module 2820, updating of the game move probability distributions p1-p3is synchronized to a time period. As previously described with respect to the game 2700, the functions of the learning module 1510 can be entirely centralized within the server 3150 or portions thereof can be distributed amongst the user computers 3110(1)-(6). When updating each of the game move probability distributions p1-p3, the game program 3200 may employ, e.g., the unweighted P-type MIMO learning methodology defined by equation [36] or the weighted P-type MIMO learning methodology defined by equation [37].
[0671] The steps performed by the game program 3200 are similar to that described with respect to FIG. 39, with the exception that the game program 3200 will independently perform the steps of the flow diagram for each of the sets of game players 3115(1)-(2), 3115(3)-(4), and 3115(5)-(6). For example, the game program 3200 will execute one pass through the flow for game players 3115(1)-(2) (and thus the first probability distribution p1), then one pass through the flow for game players 3115(3)-(4) (and thus the second probability distribution p2), and then one pass through the flow for game players 3115(5)-(6) (and thus the third probability distribution p3).
[0672] Alternatively, the game program 3200 can combine the steps of the flow diagram for the game players 3115(1)-(6). For example, referring to FIG. 49, the probability update module 3220 will first initialize the game move probability distributions p1-p3 and current player moves &lgr;2x1-&lgr;2x6, (step 3305) similarly to that described in step 405 of FIG. 9. Then, the game move selection module 3225 determines whether any of the player moves &lgr;2x1-&lgr;2x6 have been performed, and specifically whether the guns 3125(1)-(6) have been fired (step 3310). If any of player moves &lgr;2x1-&lgr;2x6 have been performed, the outcome evaluation module 3230 generates the corresponding outcome values &bgr;1-&bgr;6 for the performed ones of the player moves &lgr;x1-&lgr;x6 and corresponding game moves &agr;x1-&agr;x6 (step 3315). For each set of player moves &lgr;2x1-&lgr;2x2, &lgr;2x3&lgr;2x4, and &lgr;2x5-&lgr;2x6, the corresponding outcome values &bgr;1-&bgr;2, &bgr;3-&bgr;4, and &bgr;5-&bgr;6 can be represented by different sets of s(k), r(k) and m values (unweighted case) or ISq and IFq occurrences (weighted case). The intuition module 3215 then updates the corresponding player scores 3160(1)-(6) and duck scores 3165(1)-(6) based on the outcome values &bgr;1-&bgr;6 (step 3320), similarly to that described in steps 415 and 420 of FIG. 9. The intuition module 3215 then determines if the given time period to which the player moves &lgr;2x1-&lgr;2x6 are synchronized has expired (step 3321). If the time period has not expired, the game program 3200 will return to step 3310 where the game move selection module 3225 determines again if any of the player moves &lgr;2x1-&lgr;2x6 have been performed. If the time period has expired, the probability update module 3220 then, using the unweighted MIMO equation [36] or the weighted MIMO equation [37], updates the game move probability distributions p1-p3 based on the respective outcome values &bgr;1-&bgr;2, &bgr;3-&bgr;4, and &bgr;5-&bgr;6 (step 3325). Alternatively, rather than synchronize the asynchronous performance of the player 6 moves &lgr;2x1-&lgr;2x6 to the time period at step 3321, the probability update module 3220 can update the pertinent one of the game move probability distribution p1-p3 after each of the asynchronous player moves &lgr;2x1-&lgr;2x6 is performed using any of the techniques described with respect to the game program 300.
[0673] After step 3325, or if none of the player moves &lgr;2x1-&lgr;2x6 has been performed at step 3310, the game move selection module 3225 determines if any of the player moves have been performed, i.e., guns 3125(1)-(6), have breached the gun detection regions 3170(1)-(6) (step 3330). If none of the guns 3125(1)-(6) have breached the gun detection regions 3170(1)-(6), the game move selection module 3225 does not select any of the game moves &agr;x1-&agr;x6 from the respective game move sets &agr;1-&agr;6, and the ducks 3120(1)-(6) remain in the same location (step 3335). Alternatively, the game moves &agr;x1-&agr;x6 may be randomly selected, respectively allowing the ducks 3120(1)-(6) to dynamically wander. The game program 3200 then returns to step 3310 where it is again determined if any of the player moves &lgr;1x1-&lgr;1x6 have been performed. If any of the guns 3125(1)-(6) have breached the gun detection regions 3170(1)-(6) at step 3330, the intuition module 3215 modifies the functionality of the game move selection module 3225, and the game move selection module 3225 selects the game moves &agr;x1-&agr;x2, &agr;x3-&agr;x4, and &agr;x5-&agr;x6 from the game move sets &agr;1-&agr;2, &agr;3-&agr;4, and &agr;5-&agr;6 that correspond to the breaching guns 3125(1)-(2), 3125(3)-(4), and 3125(5)-(6) based on the corresponding performance indexes &phgr;1-&phgr;6 in the manner previously described with respect to steps 440-470 of FIG. 9 (step 3340).
[0674] It should be noted that, rather than use the game move subset selection technique, other afore-described techniques used to dynamically and continuously match the skill level of the players 3115(1)-(6) with the skill level of the game 3100, such as that illustrated in FIG. 10, can be alternatively or optionally be used as well in the game. It should also be noted that, as described with respect to FIGS. 40-44, the various modules can be distributed amongst the user computers 3110(1)-(3) and the server 3150 in a manner that optimally distributes the processing power.
[0675] Multiple-User Phone Listing Program with Multiple Probability Distributions
[0676] The learning program 3000 has other applications besides game programs. For example, the listing program 1200, which was previously described as being self-contained in the mobile phone 1100, can be distributed amongst several components or can be contained in a component separate from the mobile phone 1100 to service several phone users. Referring to FIG. 50, a priority listing program 3400 (shown in FIG. 51) is stored in a base station 1101, which services several mobile phones 1100(1)-(3) (three shown here) via respective wireless links 1 103(1)-(3). The listing program 3400 is similar to the previously described listing program 1200, with the exception that it can generate a favorite phone number list for several mobile phones 1100(1)-(3).
[0677] Referring further to FIG. 51, the listing program 3400 generally includes a probabilistic learning module 3410 and an intuition module 3415. The probabilistic learning module 3410 comprises a probability update module 3420, a phone number selection module 3425, and an outcome evaluation module 3430. Specifically, the probability update module 3420 is mainly responsible for learning each of the phone users' 11 15(1)-(3) calling habits and updating comprehensive phone number lists &agr;1-&agr;3 using probability distributions p1-p3 that, for each of the users' 11 15(1)-(3), place phone numbers in the order that they are likely to be called in the future during any given time period. The outcome evaluation module 3430 is responsible for evaluating each of the comprehensive phone number lists &agr;1-&agr;3 relative to current phone numbers &lgr;x1-&lgr;x3 called by the phone users 11 15(1)-(3).
[0678] The base station 1101 obtains the called phone numbers &lgr;x1-&lgr;x3 when the mobile phones 1100(1)-(3) place phone calls to the base station 1101 via the wireless links 1103(1)-(3). The phone number selection module 3425 is mainly responsible for selecting phone number subsets &agr;s1-&agr;s3 from the respective comprehensive phone number lists &agr;1-&agr;3 for eventual display to the phone users 1115(1)-(3) as favorite phone number lists. These phone number subsets &agr;s1-&agr;s1; are wirelessly transmitted to the respective mobile phones 1100(1)-(3) via the wireless links 1103(1)-(3) when the phone calls are established. The intuition module 3415 is responsible for directing the learning of the listing program 3400 towards the objective, and specifically, displaying the favorite phone number lists that are likely to include the phone users' 1115(1)-(3) next called phone numbers. The intuition module 3415 accomplishes this based on respective performance indexes &phgr;1-&phgr;3 (and in this case, instantaneous performance indexes &phgr;1-&phgr;3 represented as respective outcome values &bgr;1-&bgr;3).
[0679] It should be noted that the listing program 3400 can process the called phone numbers &lgr;x1-&lgr;x3 on an individual basis, resulting in the generation and transmission of respective phone number subsets &agr;s1-&agr;s3 to the mobile phones 1100(1)-(3) in response thereto, or optionally to minimize processing time, the listing program 3400 can process the called phone numbers &lgr;x1-&lgr;x3 in a batch mode, which may result in the periodic (e.g., once a day) generation and transmission of respective phone number subsets &agr;s1-&agr;s3 to the mobile phones 1100(1)-(3). In the batch mode, the phone number subsets &agr;s1-&agr;s3 can be transmitted to the respective mobile phones 1100(1)-(3) during the next phone calls from the mobile phones 1 100(1)-(3). The detailed operation of the listing program 3400 modules have previously been described, and will therefore not be reiterated here for purposes of brevity. It should also be noted that all of the processing need not be located in the base station 1101, and certain modules of the program 1200 can be located within the mobile phones 1100(1)-(3).
[0680] As will be appreciated, the phone need not be a mobile phone, but can be any phone or device that can display phone numbers to a phone user. The present invention particularly lends itself to use with mobile phones, however, because they are generally more complicated and include many more features than standard phones. In addition, mobile phone users are generally more busy and pressed for time and may not have the external resources, e.g., a phone book, that are otherwise available to phone users of home phone users. Thus, mobile phone users generally must rely on information contained in the mobile phone itself. As such, a phone that learns the phone user's habits, e.g., the phone user's calling pattern, becomes more significant in the mobile context.
[0681] Multiple-User Television Channel Listing Program with Multiple Probability Distributions
[0682] The learning program 3000 can be applied to remote controls as well. Referring now to FIG. 52, another priority listing program 3600 (shown in FIG. 53) developed in accordance with the present inventions is described in the context of another television remote control 3500. The remote control 3500 is similar to the previously described remote control 1600 with the exception that it comprises a keypad 3520 that alternatively or optionally contains multi-user keys 3565(1)-(4) respectively referred to as “FATHER”, “MOTHER”, “TEENAGE”, and “KID” keys. Alternatively, the family member keys 3565(1)-(4) can be respectively labeled “USER1,” “USER2,” “USER3,” and “USER4” keys. Operation of the multi-user keys 3565(1)-(4) allows the remote control 3500 to identify the specific person that is currently watching the television, thereby allowing it to more efficiently and accurately anticipate the television channels that the person would likely watch. Thus, each time the user operates the remote control 3500, he or she will preferably depress the corresponding multi-user key 3565 to indicate to the remote control 3500 that the user is the father, mother, teenager, or child, or some other classified user. In this manner, the remote control 3500 will be able to learn that specific user's channel watching patterns and anticipate that user's favorite television channels.
[0683] To this end, the program 3600 dynamically updates a plurality of comprehensive television channel lists for the multiple users. The comprehensive television channel lists are identical to the single comprehensive television channel list described with respect to the program 1700, with the exception that the comprehensive television channel lists are arranged and updated in such a manner that a selected one will be able to be matched with the current user 1615 and applied to the channel watching pattern of that user 1615. Alternatively, a single comprehensive television channel list is updated, and the information contained therein is extracted and stored in multiple television channel lists for the users. In this case, programming information, such as channel type, will be used to determine which television channel list the extracted information will be stored in. For example, if the channel type is “cartoons” the extracted information may be stored in the television channel list corresponding to a kid.
[0684] The listing program 3600 uses the existence or non-existence of watched television channels on the comprehensive television channel lists as performance indexes &phgr;1-&phgr;4 in measuring its performance in relation to its objective of ensuring that the comprehensive channel lists will include the future watched television channels. In this regard, it can be said that the performance indexes &phgr;1-&phgr;4 are instantaneous. Alternatively or optionally, the listing program 3600 can also use the location of the television channel on the comprehensive channel list as performance indexes &phgr;1-&phgr;4.
[0685] Referring now to FIG. 53, the listing program 3600 includes a probabilistic learning module 3610 and an intuition module 3615, which are specifically tailored for the remote control 3500. The probabilistic learning module 3610 comprises a probability update module 3620, a television channel selection module 3625, and an outcome evaluation module 3630.
[0686] Specifically, the probability update module 3620 is mainly responsible for learning the remote control users' 1615(1)-(4) television watching habits and respectively updating comprehensive television channel lists &agr;1-&agr;4 that place television channels &agr;x in the order that they are likely to be watched by the users 1615(1)-(4) in the future during any given time period. The outcome evaluation module 3630 is responsible for evaluating the comprehensive channel lists &agr;1-&agr;4 relative to current television channels &lgr;x1-&lgr;x4 watched by the respective remote control users 1615(1)-(4). The channel selection module 3625 is mainly responsible for selecting a television channel &agr;x from the comprehensive channel list &agr; corresponding to the current user 1615 upon operation of the favorite television channel key 1965.
[0687] The intuition module 3615 is responsible for directing the learning of the listing program 3600 towards the objective of selecting the television channel &agr;x that is likely to be the current remote control user's 1615 next watched television channel &agr;x. In this case, the intuition module 3615 selects the pertinent comprehensive channel list &agr;, and operates on the probability update module 3620, the details of which will be described in further detail below.
[0688] To this end, the channel selection module 3625 is configured to receive multiple television channel probability distributions p1-p4 from the probability update module 3620. Based on the television channel probability distributions p1-p4, the channel selection module 3625 generates the comprehensive channel lists &agr;1-&agr;4, each of which contains the listed television channels &agr;x ordered in accordance with their associated probability values px Thus, each comprehensive channel list &agr; contains all television channels &agr;x watched by the corresponding user 1615. From the comprehensive channel lists &agr;1-&agr;4, the channel selection module 3625 selects the list corresponding to the current user 1615, and then selects, from that list, a television channel &agr;x that the television will be switched to in the same manner that the previously described channel selection module 1725 selects a television channel from a single comprehensive television list.
[0689] Alternatively or in addition to the favorite channel key 1665, which switches the television to the next channel based on a user's generalized channel watching pattern, the keypad 3520 can include the specialized favorite channel key 1965, which switches the television to the next channel based on a user's specialized channel watching patterns. In this case, the program 3600 will operate on a plurality of linked comprehensive television channel lists &agr;1-&agr;m for each of the users 2715(1)-(4).
[0690] The outcome evaluation module 3630 is configured to receive watched television channels &lgr;x1-&lgr;x4 from the remote control users 1615(1)-(4) via the keypad 3520 using any one of the previously described methods. The outcome evaluation module 3630 is further configured to determine and output outcome values &bgr;1-&bgr;4 that indicate if the currently watched television channels &lgr;x1-&lgr;x4 respectively match television channels &agr;x1-&agr;x4 on the comprehensive channel lists &agr;1-&agr;4.
[0691] The intuition module 3615 is configured to receive the outcome value &bgr;1-&bgr;4 from the outcome evaluation module 3630 and modify the probability update module 3620, and specifically, the television channel probability distributions p1-p4. This is accomplished in the same manner as that described with respect to the intuition module 1715 when modifying the single television channel probability distribution p.
[0692] Having now described the structure of the listing program 3600, the steps performed by the listing program 3600 will be described with reference to FIG. 54. First, the outcome evaluation module 3630 determines whether one of the television channels &lgr;x1-&lgr;x4 has been newly watched (step 3705). The specific television channel watched will be specified by which multi-user key 3565 is operated. For example, (1) if multi-user key 3565(1) is operated, a currently watched television channel will be television channel &lgr;x2; (2) if multi-user key 3565(2) is operated, a currently watched television channel will be television channel &lgr;x2; (3) if multi-user key 3565(3) is operated, a currently watched television channel will be television channel &lgr;x3; and (4) if multi-user key 3565(4) is operated, a currently watched television channel will be television channel &lgr;x4.
[0693] If one of the television channels &lgr;x1-&lgr;x4 have been newly watched, the outcome evaluation module 3630 determines whether it matches a television channel &agr;x on the corresponding one of the comprehensive channel lists &agr;1-&agr;4 and generates the respective one of the outcome values &bgr;1-&bgr;4 in response thereto (step 3715). If so (&bgr;=1), the intuition module 3615 directs the probability update module 3620 to update the respective one of the television channel probability distributions p1-p4 using a learning methodology to increase the probability value px corresponding to the listed television channel &agr;x (step 3725). If not &bgr;=0), the intuition module 3615 generates a corresponding television channel &agr;x and assigns a probability value px to it, in effect, adding it to the respective one of the comprehensive channel lists &agr;1-&agr;4 (step 3730). The channel selection module 3625 then reorders the respective one of the comprehensive channel list &agr;1-&agr;4 (step 3735), sets the channel list pointer to “1” (step 3740), and returns to step 3705.
[0694] If none of the television channels &lgr;x1-&lgr;x4 has been newly watched at step 3705, e.g., if the predetermined period of time has not expired, the channel selection module 3625 determines whether the favorite channel key 1665 has been operated (step 3745). If so, the channel selection module 3625 selects a listed television channel &agr;x from one of the comprehensive channel lists 60 1-&agr;4, and in this case, the listed television channel &agr;x corresponding to the channel list pointer (step 3750). The comprehensive channel list from which the listed television channel &agr;x is selected will be specified by which multi-user key 3565 is operated. The television is then switched to the selected television channel &agr;x (step 3755), and the channel list pointer is incremented (step 3760). After step 3760, or if the favorite channel key 1665 has not been operated at step 3745, the listing program 3600 then returns to step 3705, where it is determined again if one of the television channels &lgr;x1-&lgr;x4 has been watched.
[0695] Generalized Multi-User Learning Program (Single Processor Action-Maximum Probability of Majority Approval)
[0696] Referring to FIG. 55, still another multi-user learning program 3800 developed in accordance with the present inventions can be generally implemented to provide intuitive learning capability to any variety of processing devices. In the previous multiple user action embodiments, each user action incrementally affected the relevant action probability distribution. The learning program 3800 is similar to the SIMO-based program 2200 in that multiple users 3805(1)-(3) (here, three) interact with the program 3800 by receiving the same processor action &agr;i from a processor action set &agr; within the program 3800, and each independently select corresponding user actions &lgr;x1-&lgr;x3 from respective user action sets &lgr;1-&lgr;3 based on the received processor action &agr;x. Again, in alternative embodiments, the users 3805 need not receive the processor action &agr;x, the selected user actions &lgr;x1-&lgr;x3 need not be based on the received processor action &agr;x, and/or the processor actions &agr;x may be selected in response to the selected user actions &lgr;x1-&lgr;x3. The significance is that a processor action &agr;x and user actions &lgr;x1-&lgr;x3 are selected.
[0697] The program 3800 is capable of learning based on the measured success ratio (e.g., minority, majority, super majority, unanimity) of the selected processor action &agr;x relative to the selected user actions &lgr;x1-&lgr;x3 as compared to a reference success ratio, which for the purposes of this specification, can be measured as a single outcome value 62 majx. In essence, the selected user actions &lgr;x1-&lgr;x3 are treated as a selected action vector &lgr;v. For example, if the reference success ratio for the selected processor action &agr;x is a majority, &bgr;majx may equal “1” (indicating a success) if the selected processor action &agr;x is successful relative to two or more of the three selected user actions &lgr;x1-&lgr;x3, and may equal “0” (indicating a failure) if the selected processor action &agr;x is successful relative to one or none of the three selected user actions &lgr;x1-&lgr;x3. It should be noted that the methodology contemplated by the program 3800 can be applied to a single user that selects multiple user actions to the extent that the multiple actions can be represented as an action vector &lgr;v, in which case, the determination of the outcome value &bgr;majx can be performed in the same manner. As will be described in further detail below, the program 3800 directs its learning capability by dynamically modifying the model that it uses to learn based on a performance index &phgr; to achieve one or more objectives.
[0698] To this end, the program 3800 generally includes a probabilistic learning module 3810 and an intuition module 3815. The probabilistic learning module 3810 includes a probability update module 3820, an action selection module 3825, and an outcome evaluation module 3830. Briefly, the probability update module 3820 uses learning automata theory as its learning mechanism, and is configured to generate and update an action probability distribution p based on the outcome value &bgr;majx. In this scenario, the probability update module 3820 uses a single stochastic learning automaton with a single input to a single-teacher environment (with the users 3805(1)-(3), in combination, as a single teacher), or alternatively, a single stochastic learning automaton with a single input to a single-teacher environment with multiple outputs that are treated as a single output), and thus, a SISO model is assumed. The significance is that multiple outputs, which are generated by multiple users or a single user, are quantified by a single outcome value &bgr;majx. Alternatively, if the users 3805(1)-(3) receive multiple processor actions &agr;x, some of which are different, multiple SISO models can be assumed. For example if three users receive processor action &agr;1, and two users receive processor action a2, the action probability distribution p can be sequentially updated based on the processor action &agr;1, and then updated based on the processor action &agr;2, or updated in parallel, or in combination thereof. Exemplary equations that can be used for the SISO model will be described in further detail below.
[0699] The action selection module 3825 is configured to select the processor action &agr;x from the processor action set &agr; based on the probability values px contained within the action probability distribution p internally generated and updated in the probability update module 3820. The outcome evaluation module 3830 is configured to determine and generate the outcome value &bgr;majx based on the relationship between the selected processor action &agr;x and the user action vector &lgr;v. The intuition module 3815 modifies the probabilistic learning module 3810 (e.g., selecting or modifying parameters of algorithms used in learning module 3810) based on one or more generated performance indexes &phgr; to achieve one or more objectives. As previously discussed with respect to the outcome value &bgr;, the performance index &phgr; can be generated directly from the outcome value &bgr;majx or from something dependent on the outcome value &bgr;majx, e.g., the action probability distribution p, in which case the performance index &phgr; may be a function of the action probability distribution p, or the action probability distribution p may be used as the performance index &phgr;. Alternatively, the intuition module 3815 may be non-existent, or may desire not to modify the probabilistic learning module 3810 depending on the objective of the program 3800.
[0700] The modification of the probabilistic learning module 3810 is generally accomplished similarly to that described with respect to the afore-described probabilistic learning module 110. That is, the functionalities of (1) the probability update module 3820 (e.g., by selecting from a plurality of algorithms used by the probability update module 3820, modifying one or more parameters within an algorithm used by the probability update module 3820, transforming or otherwise modifying the action probability distribution p); (2) the action selection module 3825 (e.g., limiting or expanding selection of the action &agr;x corresponding to a subset of probability values contained within the action probability distribution p); and/or (3) the outcome evaluation module 3830 (e.g., modifying the nature of the outcome value &bgr;majx or otherwise the algorithms used to determine the outcome values &bgr;majx), are modified. Specific to the learning program 3800, the intuition module 3815 may modify the outcome evaluation module 3830 by modifying the reference success ratio of the selected processor action &agr;x. For example, for an outcome value &bgr;majx to indicate a success, the intuition module 3815 may modify the reference success ratio of the selected processor action &agr;x from, e.g., a super-majority to a simple majority, or vice versa.
[0701] The various different types of learning methodologies previously described herein can be applied to the probabilistic learning module 3810. The operation of the program 3800 is similar to that of the program 2200 described with respect to FIG. 31, with the exception that, rather than updating the action probability distribution p based on several outcome values &bgr;1-&bgr;3 for the users 3805, the program 3800 updates the action probability distribution p based on a single outcome value &bgr;majx derived from the measured success of the selected processor action &agr;x relative to the selected user actions &lgr;x1-&lgr;x3, as compared to a reference success ratio. Specifically, referring to FIG. 56, the probability update module 3820 initializes the action probability distribution p (step 3850) similarly to that described with respect to step 150 of FIG. 4. The action selection module 3825 then determines if one or more of the user actions &lgr;x1-&lgr;x3 have been selected from the respective user action sets &lgr;1-&lgr;3 (step 3855). If not, the program 3800 does not select a processor action &agr;x from the processor action set &agr; (step 3860), or alternatively selects a processor action &agr;i, e.g., randomly, notwithstanding that none of the user actions &lgr;x1-&lgr;x3 has been selected (step 3865), and then returns to step 3855 where it again determines if one or more of the user actions &lgr;x1-&lgr;x3 have been selected. If one or more of the user actions &lgr;x1-&lgr;x3 have been performed at step 3855, the action selection module 3825 determines the nature of the selected ones of the user actions &lgr;x1-&lgr;x3.
[0702] Specifically, the action selection module 3825 determines whether any of the selected ones of the user actions &lgr;x1-&lgr;x3 should be countered with a processor action &agr;x (step 3870). If so, the action selection module 3825 selects a processor action &agr;x from the processor action set a based on the action probability distribution p (step 3875). After the performance of step 3875 or if the action selection module 3825 determines that none of the selected user actions &lgr;x1-&lgr;x3 is of the type that should be countered with a processor action &agr;x, the action selection module 3825 determines if any of the selected user actions &lgr;x1-&lgr;x3 are of the type that the performance index &phgr; is based on (step 3880).
[0703] If not, the program 3800 returns to step 3855 to determine again whether any of the user actions &lgr;x1-&lgr;x3 have been selected. If so, the outcome evaluation module 3830 quantifies the performance of the previously selected processor action &agr;x relative to the reference success ratio (minority, majority, supermajority, etc.) by generating a single outcome value &bgr;majx (step 3885). The intuition module 3815 then updates the performance index &phgr; based on the outcome value &bgr;majx, unless the performance index &phgr; is an instantaneous performance index that is represented by the outcome value &bgr;majx itself (step 3890). The intuition module 3815 then modifies the probabilistic learning module 3810 by modifying the functionalities of the probability update module 3820, action selection module 3825, or outcome evaluation module 3830 (step 3895). The probability update module 3820 then, using any of the updating techniques described herein, updates the action probability distribution p based on the generated outcome value &bgr;majx (step 3898).
[0704] The program 3800 then returns to step 3855 to determine again whether any of the user actions &lgr;x1-&lgr;x3 have been selected. It should be noted that the order of the steps described in FIG. 56 may vary depending on the specific application of the program 3800.
[0705] Multi-Player Game Program (Single Game Move—Maximum Probability of Majority Approval)
[0706] Having now generally described the components and functionality of the learning program 3800, we now describe one of its various applications. Referring to FIG. 57, a multiple-player game program 3900 developed in accordance with the present inventions is described in the context of the previously described duck hunting game 2300 (see FIG. 32). Because the game program 3900 will determine the success or failure of a selected game move based on the player moves as a group, in this version of the duck hunting game 2300, the players 2315(1)-(3) play against the duck 2320 as a team, such that there is only one player score 2360 and duck score 2365 that is identically displayed on all three computers 760(1)-(3).
[0707] The game program 3900 generally includes a probabilistic learning module 3910 and an intuition module 3915, which are specifically tailored for the game 2300. The probabilistic learning module 3910 comprises a probability update module 3920, a game move selection module 3925, and an outcome evaluation module 3930, which are similar to the previously described probability update module 2420, game move selection module 2425, and outcome evaluation module 2430, with the exception that they operate on the player moves &lgr;2x1-&lgr;2x3 as a player move vector &lgr;2v and determine and output a single outcome value &bgr;majx that indicates how favorable the selected game move &agr;x in comparison with the received player move vector &lgr;2v.
[0708] As previously discussed, the game move probability distribution p is updated periodically, e.g., every second, during which each of any number of the players 2315(1)-(3) may provide a corresponding number of player moves &lgr;2x1-&lgr;2x3 so that the player moves &lgr;2x1-&lgr;2x3 asynchronously performed by the players 2315(1)-(3) may be synchronized to a time period as a single player move vector &lgr;2v. It should be noted that in other types of games, where the player moves &lgr;2x need not be synchronized to a time period, such as, e.g., strategy games, the game move probability distribution p may be updated after all players have performed a player move &lgr;2x.
[0709] The game program 3900 may employ the following P-type Maximum Probability Majority Approval (MPMA) SISO equations: 44 p i ( k + 1 ) = p i ( k ) + ∑ j = 1 j ≠ i n g j ( p ( k ) ) ; and [ 38 ] pj(k+1)=pj(k)−gj(p(k)), when &bgr;majx(k)=1 and &agr;x is selected [39] 45 p i ( k + 1 ) = p i ( k ) - ∑ j = 1 j ≠ i n h j ( p ( k ) ) ; and [ 40 ] pj(k+1)=pj(k)+hj(p(k)), when, &bgr;majx(k)=0 and &agr;x is selected [40]
[0710] where px(k+1), px(k), gj(k)), hj(p(k)), i, j, k, and n have been previously defined, and ,(k) is the outcome value based on a majority success ratio of the participating players.
[0711] As an example, if there are a total of ten players, seven of which have been determined to be participating, and if two of the participating players shoot the duck 2320 and the other five participating players miss the duck 2320, &bgr;majx(k)=1, since a majority of the participating players missed the duck 2320. If, on the hand, four of the participating players shoot the duck 2320 and the other three participating players miss the duck 2320, &bgr;majx(k)=0, since a majority of the participating players hit the duck 2320. Of course, the outcome value &bgr;majx need not be based on a simple majority, but can be based on a minority, supermajority, unanimity, or equality of the participating players. In addition, the players can be weighted, such that, for any given player move &lgr;2x, a single player may be treated as two, three, or more players when determining if the success ratio has been achieved. It should be noted that a single player may perform more than one player move &lgr;2x in a single probability distribution updating time period, and thus be counted as multiple participating players. Thus, if there are three players, more than three participating players may be considered in equation.
[0712] Having now described the structure of the game program 3900, the steps performed by the game program 3900 will be described with reference to FIG. 58. First, the probability update module 3920 initializes the game move probability distribution p and current game move &agr;x (step 4005) similarly to that described in step 405 of FIG. 9. Then, the game move selection module 3925 determines whether any of the player moves &lgr;2x1-&lgr;2x3 have been performed, and specifically whether the guns 2325(1)-(3) have been fired (step 4010). If any of the player moves &lgr;2x1-&lgr;2x3 have been performed, the outcome evaluation module 3930 determines the success or failure of the currently selected game move &agr;x relative to the performed ones of the player moves &lgr;2x1-&lgr;2x3 (step 4015). The intuition module 3915 then determines if the given time period to which the player moves &lgr;2x1-&lgr;2x3 are synchronized has expired (step 4020). If the time period has not expired, the game program 3900 will return to step 4010 where the game move selection module 3925 determines again if any of the player moves &lgr;2x1-&lgr;2x3 have been performed. If the time period has expired, the outcome evaluation module 3930 determines the outcome value &bgr;majx for the player moves &lgr;2x1-&lgr;2x3, i.e., the player move vector &lgr;2v (step 4025). The intuition module 3915 then updates the combined player score 2360 and duck scores 2365 based on the outcome value &bgr;majx (step 4030). The probability update module 3920 then, using the MPMA SISO equations [38]-[41], updates the game move probability distribution p based on the generated outcome value &bgr;majx (step 4035).
[0713] After step 4035, or if none of the player moves &lgr;2x1-&lgr;2x3 has been performed at step 4010, the game move selection module 3925 determines if any of the player moves &lgr;1x1-&lgr;1x3 have been performed, i.e., guns 2325(1)-(3), have breached the gun detection region 270 (step 4040). If none of the guns 2325(1)-(3) has breached the gun detection region 270, the game move selection module 3925 does not select a game move &agr;x from the game move set &agr; and the duck 2320 remains in the same location (step 4045). Alternatively, the game move &agr;x may be randomly selected, allowing the duck 2320 to dynamically wander. The game program 3900 then returns to step 4010 where it is again determined if any of the player moves &lgr;1x1-&lgr;1x3 has been performed.
[0714] If any of the guns 2325(1)-(3) have breached the gun detection region 270 at step 4040, the intuition module 3915 modifies the functionality of the game move selection module 3925 based on the performance index &phgr;, and the game move selection module 3925 selects a game move &agr;x from the game move set &agr; in the manner previously described with respect to steps 440-470 of FIG. 9 (step 4050). It should be noted that, rather than use the game move subset selection technique, other afore-described techniques used to dynamically and continuously match the skill level of the players 2315(1)-(3) with the skill level of the game 2300, such as that illustrated in FIG. 10, can be alternatively or optionally be used as well in the game program 3900. Also, the intuition module 3915 may modify the functionality of the outcome evaluation module 3930 by modifying the reference success ratio of the selected game move &agr;x on which the single outcome value &bgr;majx is based.
[0715] The learning program 3800 can also be applied to single-player scenarios, such as, e.g., strategy games, where the player performs several game moves at a time. For example, referring to FIG. 59, a game program 4100 developed in accordance with the present inventions is described in the context of a war game, which can be embodied in any one of the previously described computer systems. In the war game, a player 4105 can select any one of a variety of combinations of weaponry to attack the game's defenses. For example, in the illustrated embodiment, the player 4105 may be able to select three weapons at a time, and specifically, one of two types of bombs (denoted by &lgr;11 and &lgr;12) from a bomb set &lgr;1, one of three types of guns (denoted by &lgr;21, &lgr;22, and &lgr;23) from a gun set &lgr;2, and one of two types of arrows (denoted by &lgr;31 and &lgr;32) from an arrow set &lgr;3. Thus, the selection of three weapons can be represented by weapon vector &lgr;v (&lgr;1x, &lgr;2y,and &lgr;3z) that will be treated as a single game move. Given that three weapons will be selected in combination, there will be a total of twelve weapon vectors &lgr;v available to the player 4105, as illustrated in the following Table 17. 17 TABLE 17 Exemplary Weapon Combinations for War Game &lgr;v &lgr;1x &lgr;2y &lgr;3z Bomb 1, Gun 1, Bomb 1 (&lgr;11) Gun 1 (&lgr;21) Arrow 1 (&lgr;31) Arrow 1 (&lgr;1) Bomb 1, Gun 1, Bomb 1 (&lgr;11) Gun 1 (&lgr;21) Arrow 2 (&lgr;32) Arrow 2 (&lgr;2) Bomb 1, Gun 2, Bomb 1 (&lgr;11) Gun 2 (&lgr;22) Arrow 1 (&lgr;31) Arrow 1 (&lgr;2) Bomb 1, Gun 2, Bomb 1 (&lgr;11) Gun 2 (&lgr;22) Arrow 2 (&lgr;32) Arrow 2 (&lgr;4) Bomb 1, Gun 3, Bomb 1 (&lgr;11) Gun 3 (&lgr;23) Arrow 1 (&lgr;31) Arrow 1 (&lgr;5) Bomb 1, Gun 3, Bomb 1 (&lgr;11) Gun 3 (&lgr;23) Arrow 2 (&lgr;32) Arrow 2 (&lgr;6) Bomb 2, Gun 1, Bomb 2 (&lgr;12) Gun 1 (&lgr;21) Arrow 1 (&lgr;31) Arrow 1 (&lgr;7) Bomb 2, Gun 1, Bomb 2 (&lgr;12) Gun 1 (&lgr;21) Arrow 2 (&lgr;32) Arrow 2 (&lgr;8) Bomb 2, Gun 2, Bomb 2 (&lgr;12) Gun 2 (&lgr;22) Arrow 1 (&lgr;31) Arrow 1 (&lgr;9) Bomb 2, Gun 2, Bomb 2 (&lgr;12) Gun 2 (&lgr;22) Arrow 2 (&lgr;32) Arrow 2 (&lgr;10) Bomb 2, Gun 3, Bomb 2 (&lgr;12) Gun 3 (&lgr;23) Arrow 1 (&lgr;31) Arrow 1 (&lgr;11) Bomb 2, Gun 3, Bomb 2 (&lgr;12) Gun 3 (&lgr;23) Arrow 2 (&lgr;32) Arrow 2 (&lgr;12)
[0716] An object of the game (such as a monster or warrior) may be able to select three defenses at a time, and specifically, one of two types of bomb defusers (denoted by &agr;11 and &agr;12) from a bomb defuser set &agr;1 against the player's bombs, one of three types of body armor (denoted by &agr;21, &agr;22, and &agr;23) from a body armor set &agr;2 against the players' guns, and one of two types of shields (denoted by &agr;11 and &agr;12) from a shield set &agr;3 against the players' arrows. Thus, the selection of three defenses can be represented by game move vector &agr;v (&agr;1x, &agr;2y,and &agr;3z) that will be treated as a single game move. Given that three defenses will be selected in combination, there will be a total of twelve game move vectors &agr;v available to the game, as illustrated in the following Table 18. 18 TABLE 18 Exemplary Defense Combinations for War Game &agr;v &agr;1x &agr;2y &agr;3z Defuser 1, Armor 1, Defuser 1 (&lgr;11) Armor 1 (&lgr;21) Shield 1 (&lgr;31) Shield 1 (&lgr;1) Defuser 1, Armor 1, Defuser 1 (&lgr;11) Armor 1 (&lgr;21) Shield 2 (&lgr;32) Shield 2 (&lgr;2) Defuser 1, Armor 2, Defuser 1 (&lgr;11) Armor 2 (&lgr;22) Shield 1 (&lgr;31) Shield 1 (&lgr;3) Defuser 1, Armor 2, Defuser 1 (&lgr;11) Armor 2 (&lgr;22) Shield 2 (&lgr;32) Shield 2 (&lgr;4) Defuser 1, Armor 3, Defuser 1 (&lgr;11) Armor 3 (&lgr;23) Shield 1 (&lgr;31) Shield 1 (&lgr;5) Defuser 1, Armor 3, Defuser 1 (&lgr;11) Armor 3 (&lgr;23) Shield 2 (&lgr;32) Shield 2 (&lgr;6) Defuser 2, Armor 1, Defuser 2 (&lgr;12) Armor 1 (&lgr;21) Shield 1 (&lgr;31) Shield 1 (&lgr;7) Defuser 2, Armor 1, Defuser 2 (&lgr;12) Armor 1 (&lgr;21) Shield 2 (&lgr;32) Shield 2 (&lgr;8) Defuser 2, Armor 2, Defuser 2 (&lgr;12) Armor 2 (&lgr;22) Shield 1 (&lgr;31) Shield 1 (&lgr;9) Defuser 2, Armor 2, Defuser 2 (&lgr;12) Armor 2 (&lgr;22) Shield 2 (&lgr;32) Shield 2 (&lgr;10) Defuser 2, Armor 3, Defuser 2 (&lgr;12) Armor 3 (&lgr;23) Shield 1 (&lgr;31) Shield 1 (&lgr;11) Defuser 2, Armor 3, Defuser 2 (&lgr;12) Armor 3 (&lgr;23) Shield 2 (&lgr;32) Shield 2 (&lgr;12)
[0717] The game maintains a score for the player and a score for the game. To this end, if the selected defenses &agr; of the game object fail to prevent one of the weapons &lgr; selected by the player from hitting or otherwise damaging the game object, the player score will be increased. In contrast, if the selected defenses &agr; of the game object prevent one of the weapons &lgr; selected by the player from hitting or otherwise damaging the game object, the game score will be increased. In this game, the selected defenses a of the game, as represented by the selected game move vector &agr;v will be successful if the game object is damaged by one or none of the selected weapons &lgr; (thus resulting in an increased game score), and will fail, if the game object is damaged by two or all of the selected weapons &lgr; (thus resulting in an increased player score). As previously discussed with respect to the game 200, the increase in the score can be fixed, one of a multitude of discrete values, or a value within a continuous range of values.
[0718] As will be described in further detail below, the game increases its skill level by learning the player's strategy and selecting the weapons based thereon, such that it becomes more difficult to damage the game object as the player becomes more skillful. The game optionally seeks to sustain the player's interest by challenging the player. To this end, the game continuously and dynamically matches its skill level with that of the player by selecting the weapons based on objective criteria, such as, e.g., the difference between the player and game scores. In other words, the game uses this score difference as a performance index &phgr; in measuring its performance in relation to its objective of matching its skill level with that of the game player. Alternatively, the performance index &phgr; can be a function of the game move probability distribution p.
[0719] The game program 4100 generally includes a probabilistic learning module 4110 and an intuition module 4115, which are specifically tailored for the war game. The probabilistic learning module 4110 comprises a probability update module 4120, a game move selection module 4125, and an outcome evaluation module 4130. Specifically, the probability update module 4120 is mainly responsible for learning the player's strategy and formulating a counterstrategy based thereon, with the outcome evaluation module 4130 being responsible for evaluating the selected defense vector &agr;v relative to the weapon vector &lgr;x, selected by the player 4105. The game move selection module 4125 is mainly responsible for using the updated counterstrategy to select the defenses in response to weapons selected by the game object. The intuition module 4115 is responsible for directing the learning of the game program 4100 towards the objective, and specifically, dynamically and continuously matching the skill level of the game with that of the player. In this case, the intuition module 4115 operates on the game move selection module 4125, and specifically selects the methodology that the game move selection module 4125 will use to select the defenses &agr;1x, &agr;2y, and &agr;3z from defense sets &agr;1, &agr;2, and &agr;3, i.e., one of the twelve defense vectors &agr;v. Optionally, the intuition module 4115 may operate on the outcome evaluation module 4130, e.g., by modifying the reference success ratio of the selected defense vector &agr;v, i.e., the ratio of hits to the number of weapons used. Of course if the immediate objective is to merely determine the best defense vector &agr;v, the intuition module 4115 may simply decide to not modify the functionality of any of the modules.
[0720] To this end, the outcome evaluation module 4130 is configured to receive weapons &lgr;1x, &lgr;2y, and &lgr;3z from the player, i.e., one of the twelve weapon vectors A. The outcome evaluation module 4130 then determines whether the previously selected defenses &agr;1x, &agr;2y, and &agr;3z, i.e., one of the twelve defense vectors &agr;v, were able to prevent damage incurred from the received weapons &lgr;1x, &lgr;2y, and &lgr;3z, with the outcome value , &bgr;majx equaling one of two predetermined values, e.g., “1” if two or more of the defenses &agr;1x, &agr;2y, and &agr;3z were successful, or “0” if two or more of the defenses &agr;1x, &agr;2y, and &agr;3z were unsuccessful.
[0721] The probability update module 4120 is configured to receive the outcome values &bgr;majx from the outcome evaluation module 4130 and output an updated game strategy (represented by game move probability distribution p) that the game object will use to counteract the player's strategy in the future. The probability update module 4120 updates the game move probability distribution p using the P-type MPMA SISO equations [38]-[41], with the game move probability distribution p containing twelve probability values pv corresponding to the twelve defense vectors &agr;v. The game move selection module 4125 pseudo-randomly selects the defense vector &agr;v based on the updated game strategy, and is thus, further configured to receive the game move probability distribution p from the probability update module 4120, and selecting the defense vector &agr;v based thereon.
[0722] The intuition module 4115 is configured to modify the functionality of the game move selection module 4125 based on the performance index &phgr;, and in this case, the current skill level of the players relative to the current skill level of the game. In the preferred embodiment, the performance index &phgr; is quantified in terms of the score difference value &Dgr; between the player score and the game object score. In the manner described above with respect to game 200, the intuition module 4115 is configured to modify the functionality of the game move selection module 4125 by subdividing the set of twelve defense vectors &Dgr; into a plurality of defense vector subsets, and selecting one of the defense vectors subsets based on the score difference value &Dgr;. The game move selection module 4125 is configured to pseudo-randomly select a single defense vector &agr;v from the selected defense vector subset. Alternatively, the intuition module 4115 modifies the maximum number of defenses &agr; in the defense vector &agr;v that must be successful from two to one, e.g., if the relative skill level of the game object is too high, or from two to three, e.g., if the relative skill level of the game object is too low. Even more alternatively, the intuition module 4115 does not exist or determines not to modify the functionality of any of the modules, and the game move selection module 4125 automatically selects the defense vector &agr;v corresponding to the highest probability value pv to always find the best defense for the game object.
[0723] Having now described the structure of the game program 4100, the steps performed by the game program 4100 will be described with reference to FIG. 60. First, the probability update module 4120 initializes the game move probability distributions and current defense vector &agr;v (step 4205) similarly to that described in step 405 of FIG. 9. Then, the intuition module 4115 modifies the functionality of the game move selection module 4125 based on the performance index &phgr;, and the game move selection module 4125 selects a defense vector &agr;v from the defense vector set &agr; in the manner previously described with respect to steps 440-470 of FIG. 9 (step 4210). It should be noted that, rather than use the game move subset selection technique, other afore-described techniques used to dynamically and continuously match the skill level of the player 4105 with the skill level of the game, such as that illustrated in FIG. 10, can be alternatively or optionally be used as well in the game program 4100. Also, the intuition module 4115 may modify the functionality of the outcome evaluation module 4130 by modifying the success ratio of the selected defense vector &agr;v on which the single outcome value &bgr;majx is based. Even more alternatively, the intuition module 4115 may not modify the functionalities of any of the modules, e.g., if the objective is to find the best defense vector &agr;v.
[0724] Then, the outcome evaluation module 4130 determines whether the weapon vector &agr;v has been selected (step 4215). If no weapon vector &agr;v has been selected at step 4215, the game program 4100 then returns to step 4215 where it is again determined if a weapon vector &lgr;v has been selected. If the a weapon vector &lgr;v has been selected, the outcome evaluation module 4130 then determines how many of the defenses in the previously selected defense vector &agr;v were successful against the respective weapons of the selected weapon vector &lgr;v, and generates the outcome value &bgr;majx in response thereto (step 4220). The intuition module 4115 then updates the player scores and game object score based on the outcome values &bgr;majx (step 4225). The probability update module 4120 then, using the MPMA SISO equations [38]-[41], updates the game move probability distribution pv based on the generated outcome value &bgr; (step 4230). The game program 4100 then returns to step 4210 where another defense vector &agr;v is selected.
[0725] The learning program 3800 can also be applied to the extrinsic aspects of games, e.g., revenue generation from the games. For example, referring to FIG. 61, a learning software revenue program 4300 developed in accordance with the present inventions is described in the context of an internet computer game that provides five different scenarios (e.g., forest, mountainous, arctic, ocean, and desert) with which three players 4305(1)-(3) can interact. The objective the program 4300 is to generate the maximum amount of revenue as measured by the amount of time that each player 4305 plays the computer game. The program 4300 accomplishes this by providing the players 4305 with the best or more enjoyable scenarios. Specifically, the program 4300 selects three scenarios designated from the five scenario set &agr; at time for each player 4305 to interact with. Thus, the selection of three scenarios can be represented by a scenario vector &agr;v that will be treated as a single game move. Given that three scenarios will be selected in combination from five scenarios, there will be a total of ten scenario vectors &agr;v available to the players 4305, as illustrated in the following Table 19. 19 TABLE 19 Exemplary Scenario Combinations for the Revenue Generating Computer Game &agr;v Forest, Mountainous, Arctic (&agr;1) Forest, Mountainous, Ocean (&agr;2) Forest, Mountainous, Desert (&agr;3) Forest, Arctic, Ocean (&agr;4) Forest, Arctic, Desert (&agr;5) Forest, Ocean, Desert (&agr;6) Mountainous, Arctic, Ocean (&agr;7) Mountainous, Arctic, Desert (&agr;8) Mountainous, Ocean, Desert (&agr;9) Arctic, Ocean, Desert (&agr;10)
[0726] In this game, the selected scenarios a of the game, as represented by the selected game move vector &agr;v, will be successful if two or more of the players 4305 play the game for at least a predetermined time period (e.g., 30 minutes), and will fail, if one or less of the players 4305 play the game for at least the predetermined time period. In this case, the player move A can be considered a continuous period of play. Thus, three players 4305(1)-(3) will produce three respective player moves &lgr;1-&lgr;3. The revenue program 4300 maintains a revenue score, which is a measure of the target incremental revenue with the current generated incremental revenue. The revenue program 4300 uses this revenue as a performance index &phgr; in measuring its performance in relation to its objective of generating the maximum revenue.
[0727] The revenue program 4300 generally includes a probabilistic learning module 4310 and an intuition module 4315, which are specifically tailored to obtain the maximum revenue. The probabilistic learning module 4310 comprises a probability update module 4320, a scenario selection module 4325, and an outcome evaluation module 4330. Specifically, the probability update module 4320 is mainly responsible for learning the players' 4305 favorite scenarios, with the outcome evaluation module 4330 being responsible for evaluating the selected scenario vector &agr;v relative to the favorite scenarios as measured by the amount of time that game is played. The scenario selection module 4325 is mainly responsible for using the learned scenario favorites to select the scenarios. The intuition module 4315 is responsible for directing the learning of the revenue program 4300 towards the objective, and specifically, obtaining maximum revenue. In this case, the intuition module 4315 operates on the outcome evaluation module 4330, e.g., by modifying the success ratio of the selected scenario vector &agr;x, or the time period of play that dictates the success or failure of the selected defense vector &agr;v. Alternatively, the intuition module 4315 may simply decide to not modify the functionality of any of the modules.
[0728] To this end, the outcome evaluation module 4330 is configured to player moves &lgr;1-&lgr;3 from the respective players 4305(1)-(3). The outcome evaluation module 4330 then determines whether the previously selected scenario vector &agr;v was played by the players 4305(1)-(3) for the predetermined time period, with the outcome value &lgr;majx equaling one of two predetermined values, e.g., “1” if the number of times the selected scenario vector &agr;v exceeded the predetermined time period was two or more times, or “0” if the number of times the selected scenario vector &agr;v exceeded the predetermined time period was one or zero times.
[0729] The probability update module 4320 is configured to receive the outcome values &bgr;majx from the outcome evaluation module 4330 and output an updated game strategy (represented by scenario probability distribution p) that will be used to select future scenario vectors &agr;v. The probability update module 4320 updates the scenario probability distribution p using the P-type MPMA SISO equations [38]-[41], with the scenario probability distributions containing ten probability values pv corresponding to the ten scenario vectors &agr;v. The scenario selection module 4325 pseudo-randomly selects the scenario vector &agr;v based on the updated revenue strategy, and is thus, further configured to receive the scenario probability distributions from the probability update module 4320, and selecting the scenario vector &agr;v based thereon.
[0730] The intuition module 4315 is configured to modify the functionality of the outcome evaluation module 4330 based on the performance index &phgr;, and in this case, the revenue score. The scenario selection module 4325 is configured to pseudo-randomly select a single scenario vector &agr;v from the ten scenario vectors &agr;v. For example, the intuition module 4315 can modify the maximum number of times the play time for the scenario vector &agr;x exceeds the predetermined period of time from two to one or from two to three. Even more alternatively, the intuition module 4315 does not exist or determines not to modify the functionality of any of the modules.
[0731] Having now described the structure of the game program 4300, the steps performed by the game program 4300 will be described with reference to FIG. 62. First, the probability update module 4320 initializes the scenario probability distribution p and current scenario vector &agr;v (step 4405). Then, the scenario selection module 4325 determines whether any of the player moves &lgr;1-&lgr;3 have been performed, and specifically whether play has been terminated by the players 4305(1)-(3) (step 4410). If none of the player moves &lgr;1-&lgr;3 has been performed, the program 4300 returns to step 4410 where it again determines if any of the player &lgr;1-&lgr;3 have been performed. If any of the player moves &lgr;1-&lgr;3 have been performed, the outcome evaluation module 4330 determines the success or failure of the currently selected scenario vector &agr;v relative to continuous play period corresponding to the performed ones of the player moves &lgr;1-&lgr;3, i.e., whether any of the players 4305(1)-(3) terminated play (step 4415). The intuition module 2015 then determines if all three of the player moves &lgr;1-&lgr;3 have been performed (step 4420). If not, the game program 4300 will return to step 4410 where the scenario selection module 4325 determines again if any of the player moves &lgr;1-&lgr;3 have been performed. If all three of the player moves &lgr;1-&lgr;3 have been performed, the outcome evaluation module 4330 then determines how many times the play time for the selected scenario vector &agr;v exceeded the predetermined time period, and generates the outcome value &bgr;majx in response thereto (step 4425). The probability update module 4320 then, using the MPMA SISO equations [38]-[41], updates the scenario probability distribution p based on the generated outcome value &bgr;majx (step 4430). The intuition module 3915 then updates the revenue score based on the outcome value &bgr;majx (step 4435), and then modifies the functionality of the outcome evaluation module 4330 (step 4440). The game move selection module 3925 then pseudo-randomly selects a scenario vector &agr;v (step 4445).
[0732] Generalized Multi-User Learning Program (Single Processor Action-Maximum Number of Teachers Approving)
[0733] Referring to FIG. 63, yet another multi-user learning program 4500 developed in accordance with the present inventions can be generally implemented to provide intuitive learning capability to any variety of processing devices. The learning program 4500 is similar to the program 3800 in that multiple users 4505(1)-(5) (here, five) interact with the program 4500 by receiving the same processor action &agr;x from a processor action set &agr; within the program 4500, and each independently selecting corresponding user actions &lgr;x1-&lgr;x5 from respective user action sets &lgr;1-&lgr;5 based on the received processor action &agr;x. The learning program 4500 differs from the program 3800 in that, rather than learning based on the measured success ratio of a selected processor action &agr;x relative to a reference success ratio, it learns based on whether the selected processor action &agr;x has a relative success level (in the illustrated embodiment, the greatest success) out of processor action set &agr; for the maximum number of users 4505. For example, &bgr;max may equal “1” (indicating a success) if the selected processor action &agr;x is the most successful for the maximum number of users 4505, and may equal “0” (indicating a failure) if the selected processor action &agr;x is not the most successful for the maximum number of users 4505. To determine which processor action &agr;x is the most successful, individual outcome values &bgr;1-&bgr;5 are generated and accumulated for the user actions &lgr;x1-&lgr;x5 relative to each selected action &agr;i. As will be described in further detail below, the program 4500 directs its learning capability by dynamically modifying the model that it uses to learn based on a performance index &phgr; to achieve one or more objectives.
[0734] To this end, the program 4500 generally includes a probabilistic learning module 4510 and an intuition module 4515. The probabilistic learning module 4510 includes a probability update module 4520, an action selection module 4525, and an outcome evaluation module 4530. Briefly, the probability update module 4520 uses learning automata theory as its learning mechanism, and is configured to generate and update a single action probability distribution p based on the outcome value &bgr;max. In this scenario, the probability update module 4520 uses a single stochastic learning automaton with a single input to a single-teacher environment (with the users 4505(1)-(5), in combination, as a single teacher), and thus, a SISO model is assumed. Alternatively, if the users 4505(1)-(5) receive multiple processor actions &agr;x, some of which are different, multiple SISO models can be assumed, as previously described with respect to the program 3800. Exemplary equations that can be used for the SISO model will be described in further detail below.
[0735] The action selection module 4525 is configured to select the processor action &agr;x from the processor action set &agr; based on the probability values px contained within the action probability distributions internally generated and updated in the probability update module 4520. The outcome evaluation module 4530 is configured to determine and generate the outcome values &bgr;1-&bgr;5 based on the relationship between the selected processor action &agr;x and the user actions &lgr;x1-&lgr;x5. The outcome evaluation module 4530 is also configured to determine the most successful processor action &agr;x for the maximum number of users 4505(1)-(5), and generate the outcome value &bgr;max based thereon.
[0736] The outcome evaluation module 4530 can determine the most successful processor action &agr;x for the maximum number of users 4505(1)-(5) by reference to action probability distributions p1-p5 maintained for the respective users 4505(1)-(5). Notably, these action probability distributions p1-p5 would be updated and maintained using the SISO model, while the single action probability distribution p described above will be separately updated and maintained using a Maximum Number of Teachers Approving (MNTA) model, which uses the outcome value &bgr;max. For example, Table 20 illustrates exemplary probability distributions p1-p5 for the users 4505(1)-(5), with each of the probability distributions p1-p5 having seven probability values px corresponding to seven processor actions &agr;x. As shown, the highest probability values, and thus, the most successful processor actions &agr;x for the respective users 4505(1)-(5), are &agr;4 (p4=0.92) for user 4505(1), &agr;5 (p1=0.93) for user 4505(2), &agr;4 (p4=0.94) for user 4505(3), &agr;4 (p4=0.69) for user 4505(4), and &agr;4 (p7=0.84) for user 4505(5). Thus, for the exemplary action probability distributions p shown in Table 20, the most successful processor action &agr;x for the maximum number of users 4505(1)-(5) (in this case, users 4505(1), 4505(3), and 4505(4)) will be processor action &agr;4, and thus, if the action selected is &agr;4, &bgr;max will equal “1”, resulting an increase in the action probability value p4, and if the action selected is other than &agr;4, &bgr;max will equal “0”, resulting in a decrease in the action probability value p4.
[0737] The outcome evaluation module 4530 can also determine the most successful processor action &agr;x for the maximum number of users 4505(1)-(5) by generating and maintaining an estimator table of the successes and failures of each of the processor action &agr;x relative to the user actions user actions &lgr;x1-&lgr;x5. This is actually the preferred method, since it will more quickly converge to the most successful processor action &agr;x for any given user 4505, and requires less processing power. For example, Table 21 illustrates exemplary success to total number ratios rx for each of the seven processor actions &agr;x and for each of the users 4505(1)-(5). As shown, the highest probability values, and thus, the most successful processor actions &agr;x for the respective users 4505(1)-(5), are &agr;4 (r4=⅘) for user 4505(1), &agr;6 (r6={fraction (9/10)}) for user 4505(2), &agr;4 (r6={fraction (8/10)}) for user 4505(3), &agr;7 (r7={fraction (6/7)}) for user 4505(4), and &agr;2 (r2=⅚) for user 4505(5). Thus, for the exemplary success to total number ratios r shown in Table 21, the most successful processor action &agr;x for the maximum number of users 4505(1)-(5) (in this case, users 4505(2) and 3205(3)) will be processor action &agr;6, and thus, if the action selected is &agr;6, &bgr;max will equal “1”, resulting an increase in the action probability value p6 for the single action probability distribution p, and if the action selected is other than &agr;6, &bgr;max will equal “0”, resulting in a decrease in the action probability value p6 for the single action probability distribution p. 20 TABLE 21 Exemplary Estimator Table For Five Users r r1 r2 r3 r4 r5 r6 r7 1 3/10 2/6 9/12 4/5 2/9 4/10 4/7 2 6/10 4/6 4/12 3/5 4/9 9/10 5/7 3 7/10 3/6 8/12 2/5 6/9 8/10 3/7 4 5/10 4/6 2/12 4/5 5/9 6/10 6/7 5 3/10 5/6 6/12 3/5 2/9 5/10 4/7
[0738] The intuition module 4515 modifies the probabilistic learning module 4510 (e.g., selecting or modifying parameters of algorithms used in learning module 4510) based on one or more generated performance indexes &phgr; to achieve one or more objectives. As previously discussed, the performance index &phgr; can be generated directly from the outcome values &bgr;1-&bgr;5 or from something dependent on the outcome values &bgr;1-&bgr;5, e.g., the action probability distributions p1-p5, in which case the performance index &phgr; may be a function of the action probability distributions p1-p5, or the action probability distributions p1-p5 may be used as the performance index &phgr;. Alternatively, the intuition module 4515 may be non-existent, or may desire not to modify the probabilistic learning module 4510 depending on the objective of the program 4500.
[0739] The modification of the probabilistic learning module 4510 is generally accomplished similarly to that described with respect to the afore-described probabilistic learning module 110. That is, the functionalities of (1) the probability update module 4520 (e.g., by selecting from a plurality of algorithms used by the probability update module 4520, modifying one or more parameters within an algorithm used by the probability update module 4520, transforming or otherwise modifying the action probability distribution p); (2) the action selection module 4525 (e.g., limiting or expanding selection of the action &agr;x corresponding to a subset of probability values contained within the action probability distribution p); and/or (3) the outcome evaluation module 4530 (e.g., modifying the nature of the outcome values &bgr;1-&bgr;5 or otherwise the algorithms used to determine the outcome values &bgr;1-&bgr;5), are modified. Specific to the learning program 4500, the intuition module 4515 may modify the outcome evaluation module 4530 to indicate which processor action &agr;x is the least successful or average successful processor action &agr;i for the maximum number of users 4505.
[0740] The various different types of learning methodologies previously described herein can be applied to the probabilistic learning module 4510. The operation of the program 4500 is similar to that of the program 2200 described with respect to FIG. 31, with the exception that, rather than updating the action probability distribution p based on several outcome values &bgr;1-&bgr;5 for the users 4505, the program 4500 updates the action probability distributionp based on the outcome value &bgr;max.
[0741] Specifically, referring to FIG. 64, the probability update module 4520 initializes the action probability distribution p (step 4550) similarly to that described with respect to step 150 of FIG. 4. The action selection module 4525 then determines if one or more of the users 4505(1)-(5) have selected a respective one or more of the user actions &lgr;x1-&lgr;x5 (step 4555). If not, the program 4500 does not select a processor action &agr;x from the processor action set &agr; (step 4560), or alternatively selects a processor action &agr;x, e.g., randomly, notwithstanding that none of the users 4505 has selected a user actions &lgr;x (step 4565), and then returns to step 3755 where it again determines if one or more of the users 4505 have selected the respective one or more of the user actions &lgr;x1-&lgr;x5.
[0742] If so, the action selection module 4525 determines whether any of the selected user actions &lgr;x1-&lgr;x5 should be countered with a processor action &agr;x (step 4570). If they should, the action selection module 4525 selects a processor action &agr;x from the processor action set &agr; based on the action probability distributionp (step 4575). After the selection of step 4575 or if the action selection module 4525 determines that none of the selected user actions &lgr;x1-&lgr;x5 should be countered with a processor action &agr;x, the outcome evaluation module 4530, the action selection module 4525 determines if any of the selected user actions &lgr;x1-&lgr;x5 are of the type that the performance index &phgr; is based on (step 4580).
[0743] If not, the program 4500 returns to step 4555. If so, the outcome evaluation module 4530 quantifies the selection of the previously selected processor action &agr;x relative to the selected ones of the user actions &lgr;x1-&lgr;x5 by generating the respective ones of the outcome values &bgr;1-&bgr;5 (step 4585). The probability update module 4520 then updates the individual action probability distributions p1-p5 or estimator table for the respective users 4505 (step 4590), and the outcome evaluation module 4530 then determines the most successful processor action &agr;x for the maximum number of users 4505, and generates outcome value &bgr;max (step 4595).
[0744] The intuition module 4515 then updates the performance index &phgr; based on the relevant outcome values &bgr;1-&bgr;5 unless the performance index &phgr; is an instantaneous performance index that is represented by the outcome values &bgr;1-&bgr;5 themselves (step 4596). The intuition module 4515 then modifies the probabilistic learning module 4510 by modifying the functionalities of the probability update module 4520, action selection module 4525, or outcome evaluation module 4530 (step 4597). The probability update module 4520 then, using any of the updating techniques described herein, updates the action probability distribution p based on the generated &bgr;max (step 4598).
[0745] The program 4500 then returns to step 4555 to determine again whether one or more of the users 4505(1)-(5) have selected a respective one or more of the user actions &lgr;x1-&lgr;x5 . It should be noted that the order of the steps described in FIG. 64 may vary depending on the specific application of the program 4500.
[0746] Multi-Player Game Program (Single Game Move-Maximum Number of Teachers Approving)
[0747] Having now generally described the components and functionality of the learning program 4500, we now describe one of its various applications. Referring to FIG. 65, a multiple-player game program 4600 developed in accordance with the present inventions is described in the context of the previously described duck hunting game 2300 (see FIG. 32). Because the game program 4600 will determine the success or failure of a selected game move based on the player moves as a group, in this version of the duck hunting game 2300, the players 2315(1)-(3) play against the duck 2320 as a team, such that there is only one player score 2360 and duck score 2365 that is identically displayed on all three computers 2310(1)-(3).
[0748] The game program 4600 generally includes a probabilistic learning module 4610 and an intuition module 4615, which are specifically tailored for the game 2300. The probabilistic learning module 4610 comprises a probability update module 4620, a game move selection module 4625, and an outcome evaluation module 4630, which are similar to the previously described probability update module 3920, game move selection module 3925, and outcome evaluation module 3930, with the exception that it does not operate on the player moves &lgr;2x1&lgr;2x3 as a vector, but rather generates multiple outcome values &bgr;1-&bgr;3 for the player moves &lgr;2x1-&lgr;2x3, determines the game move &agr;x that is the most successful out of game move set &agr; for the maximum number of players 2315(1)-(3), and then generates an outcome value &bgr;max.
[0749] As previously discussed, the game move probability distribution p is updated periodically, e.g., every second, during which each of any number of the players 2315(1)-(3) may provide a corresponding number of player moves &lgr;2x1-&lgr;2x3 so that the player moves &lgr;2x1-&lgr;2x3 asynchronously performed by the players 2315(1)-(3) may be synchronized to a time period. It should be noted that in other types of games, where the player moves &lgr;2x need not be synchronized to a time period, such as, e.g., strategy games, the game move probability distribution p may be updated after all players have performed a player move &lgr;2x.
[0750] The game program 4600 may employ the following P-type Maximum Number of Teachers Approving (MNTA) SISO equations: 46 p i ( k + 1 ) = p i ( k ) + ∑ j = 1 j ≠ i n g j ( p ( k ) ) ; and [ 42 ] pj(k+1)=pj(k)−gj(p(k)), when &bgr;max(k)=1 and &agr;x is selected[43] 47 p i ( k + 1 ) = p i ( k ) - ∑ j = 1 j ≠ i n h j ( p ( k ) ) ; and [ 44 ] pj(k+1)=pj(k)+hj(p(k)), when &bgr;max(k)=0 and &agr;x is selected [45]
[0751] where px(k+1), px(k), gj(p(k)), hj(p(k)), i, j, k, and n have been previously defined, and &bgr;max(k) is the outcome value based on a maximum number of the players for which the selected game move &agr;x is successful.
[0752] The game move &agr;x that is the most successful for the maximum number of players can be determined based on a cumulative success/failure analysis of the duck hits and misses relative to all of the game move &agr;x as derived from game move probability distributions p maintained for each of the players, or from the previously described estimator table. As an example, assuming the game move &agr;4 was selected and there are a total of ten players, if game move &agr;4 is the most successful for four of the players, game move &agr;1 is the most successful for three of the players, game move &agr;7 is the most successful for two of the players, and game move &agr;4 is the most successful for one of the players, &bgr;max(k)=1 since the game move &agr;4 is the most successful for the maximum number (four) of players. If, however, game move &agr;4 is the most successful for two of the players, game move &agr;1 is the most successful for three of the players, game move &agr;7 is the most successful for four of the players, and game move &agr;4 is the most successful for one of the players, &bgr;max(k)=0, since the game move &agr;4 is not the most successful for the maximum number of players.
[0753] Having now described the structure of the game program 4600, the steps performed by the game program 4600 will be described with reference to FIG. 66. First, the probability update module 4620 initializes the game move probability distribution p and current game move &agr;hd ix (step 4705) similarly to that described in step 405 of FIG. 9. Then, the game move selection module 4625 determines whether any of the player moves &lgr;2x1-&lgr;2x3 have been performed, and specifically whether the guns 2325(1)-(3) have been fired (step 4710). If any of the player moves &lgr;2x1-&lgr;2x3 have been performed, the outcome evaluation module 4630 determines the success or failure of the currently selected game move &agr;x relative to the performed ones of the player moves &lgr;2x1-&lgr;2x3 (step 4715). The intuition module 4615 then determines if the given time period to which the player moves &lgr;2x1-&lgr;2x3 are synchronized has expired (step 4720). If the time period has not expired, the game program 4600 will return to step 4710 where the game move selection module 4625 determines again if any of the player moves &lgr;2x1-&lgr;2x3 have been performed. If the time period has expired, the outcome evaluation module 4630 determines the outcome values &bgr;1-&bgr;3 for the performed one of the player moves &lgr;2x1-&lgr;2x3 (step 4725). The probability update module 4620 then updates the game move probability distributions p1-p3 for the players 2315(1)-(3) or updates the estimator table (step 4730). The outcome evaluation module 4630 then determines the most successful game move &agr;x for each of the players 2315 (based on the separate probability distributions p1-p3 or estimator table), and then generates the outcome value &bgr;max (step 4735). The intuition module 4615 then updates the combined player 2360 and duck scores 2365 based on the separate outcome values &bgr;1-&bgr;3 (step 4740). The probability update module 4620 then, using the MNTA SISO equations [42]-[45], updates the game move probability distribution p based on the generated outcome value &bgr;max (step 4745).
[0754] After step 4745, or if none of the player moves &lgr;2x1-&lgr;2x3 has been performed at step 4710, the game move selection module 4625 determines if any of the player moves &lgr;2x1-&lgr;2x3 have been performed, i.e., guns 2325(1)-(3), have breached the gun detection region 270 (step 4750). If none of the guns 2325(1)-(3) has breached the gun detection region 270, the game move selection module 4625 does not select a game move &agr;x from the game move set &agr; and the duck 2320 remains in the same location (step 4755). Alternatively, the game move &agr;x may be randomly selected, allowing the duck 2320 to dynamically wander. The game program 4600 then returns to step 4710 where it is again determined if any of the player moves &lgr;1x1-&lgr;1x3 have been performed. If any of the guns 2325(1)-(3) have breached the gun detection region 270 at step 4750, the intuition module 4615 may modify the functionality of the game move selection module 4625 based on the performance index &phgr;, and the game move selection module 4625 selects a game move &agr;x from the game move set &agr; in the manner previously described with respect to steps 440-470 of FIG. 9 (step 4760). It should be noted that, rather than use the game move subset selection technique, other afore-described techniques used to dynamically and continuously match the skill level of the players 2315(1)-(3) with the skill level of the game 2300, such as that illustrated in FIG. 10, can be alternatively or optionally be used as well in the game program 4600. Also, the intuition module 4615 may modify the functionality of the outcome evaluation module 4630 by changing the most successful game move to the least or average successful &agr;x for each of the players 2315(1)-(3).
[0755] Generalized Multi-User Learning Program (Single Processor Action-Teacher Action Pair)
[0756] Referring to FIG. 67, still another multi-user learning program 4800 developed in accordance with the present inventions can be generally implemented to provide intuitive learning capability to any variety of processing devices. Unlike the previous embodiments, the learning program 4800 may link processor actions with user parameters (such as, e.g., users or user actions) to generate action pairs, or trios or higher numbered groupings.
[0757] The learning program 4800 is similar to the SIMO-based program 2200 in that multiple users 4805(1)-(3) (here, three) interact with the program 4800 by receiving the same processor action &agr;x from a processor action set &agr; within the program 4800, each independently selecting corresponding user actions &lgr;x1-&lgr;x3 from respective user action sets &lgr;x1-&lgr;x3 based on the received processor action ax. Again, in alternative embodiments, the users 4805 need not receive the processor action &agr;x, the selected user actions &lgr;x1-&lgr;x3 need not be based on the received processor action &agr;x, and/or the processor actions &agr;x may be selected in response to the selected user actions &lgr;x1-&lgr;x3. The significance is that a processor action &agr;x and user actions &lgr;x1-&lgr;x3 are selected.
[0758] The program 4800 is capable of learning based on the measured success or failure of combination of user/processor action pairs &agr;uix, which for the purposes of this specification, can be measured as outcome values &bgr;uix, where u is the index for a specific user 4805, and i is the index for the specific processor action &agr;x. For example, if the processor action set &agr; includes seventeen processor actions &agr;x, than the number of user/processor action pairs &agr;uix will equal fifty-one (three users 4805 multiplied by seventeen processor actions &agr;x). As an example, if selected processor action &agr;8 is successful relative to a user action &lgr;x selected by the second user 4805(2), then &bgr;2,8 may equal “1” (indicating a success), and if processor action &agr;8 is not successful relative to a user action &lgr;x selected by the second user 4805(2), then &bgr;2,8 may equal “0” (indicating a failure).
[0759] It should be noted that other action pairs are contemplated. For example, instead of linking the users 4805 with the processor actions &agr;x, the user actions &lgr;x can be linked to the processor actions &agr;x, to generate user action/processor action pairs &agr;xix, which again can be measured as outcome values &bgr;xix, where i is the index for the selected action &agr;x, and x is the index for the selected action &lgr;x. For example, if the processor action set &agr; includes seventeen processor actions &agr;x, and the user action set &agr; includes ten user actions &lgr;x, than the number of user action/processor action pairs &agr;xix will equal one hundred seventy (ten user actions &lgr;x multiplied by seventeen processor actions &agr;x). As an example, if selected processor action &agr;12 is successful relative to user action &lgr;6 selected by a user 4805 (either a single user or one of a multiple of users), then &bgr;6,12 may equal “1” (indicating a success), and if selected processor action &agr;12 is not successful relative to user action 26 selected by a user 4805, then &bgr;6,12 may equal “0” (indicating a failure).
[0760] As another example, the users 4805, user actions &lgr;x, and processor actions &agr;x, can be linked together to generate user/user action/processor action trios &agr;uxix, which can be measured as outcome values &bgr;uxix, where u is the index for the user 4805, i is the index for the selected action &agr;i, and x is the index for the selected user action &lgr;x. For example, if the processor action set a includes seventeen processor actions &agr;x, and the user action set &agr; includes ten user actions &lgr;x, than the number of user/user action/processor action trios &agr;uxi will equal five hundred ten (three users 4805 multiplied by ten user actions &lgr;x multiplied by seventeen processor actions &agr;x). As an example, if selected processor action &agr;12 is successful relative to user action 26 selected by the third user 4805(3) (either a single user or one of a multiple of users), then &bgr;3,6,12 may equal “1” (indicating a success), and if selected processor action &agr;12 is not successful relative to user action 26 selected by the third user 4805(3), then &bgr;3,6,12 may equal “0” (indicating a failure).
[0761] It should be noted that the program 4800 can advantageously make use of estimator tables should the number of processor action pairs or trio become too numerous. The estimator table will keep track of the number of successes and failures for each of the action pairs or trios. In this manner, the processing required for the many processor actions pairs or trios can be minimized. The action probability distribution p can then be periodically updated based on the estimator table.
[0762] To this end, the program 4800 generally includes a probabilistic learning module 4810 and an intuition module 4815. The probabilistic learning module 4810 includes a probability update module 4820, an action selection module 4825, and an outcome evaluation module 4830. Briefly, the probability update module 4820 uses learning automata theory as its learning mechanism, and is configured to generate and update an action probability distribution p containing probability values (either puix or pxix or puxix) based on the outcome values &bgr;uix or &bgr;xix in the case of action pairs, or based on outcome values &bgr;uxix, in the case of action trios. In this scenario, the probability update module 4820 uses a single stochastic learning automaton with a single input to a single-teacher environment (with the users 4805(1)-(3), in combination, as a single teacher), or alternatively, a single stochastic learning automaton with a single input to a single-teacher environment with multiple outputs that are treated as a single output), and thus, a SISO model is assumed. The significance is that the user actions, processor actions, and/or the users are linked to generate action pairs or trios, each of which can be quantified by a single outcome value &bgr;. Exemplary equations that can be used for the SISO model will be described in further detail below.
[0763] The action selection module 4825 is configured to select the processor action &agr;x from the processor action set &agr; based on the probability values (either puix or pxix or puxix) contained within the action probability distribution p internally generated and updated in the probability update module 4820. The outcome evaluation module 4830 is configured to determine and generate the outcome value &bgr; (either &bgr;uix or &bgr;xix or &bgr;uxix) based on the relationship between the selected processor action &agr;x and the selected user action &lgr;x. The intuition module 4815 modifies the probabilistic learning module 4810 (e.g., selecting or modifying parameters of algorithms used in learning module 4810) based on one or more generated performance indexes &phgr; to achieve one or more objectives. As previously discussed, the performance index &phgr; can be generated directly from the outcome value for from something dependent on the outcome value &bgr;, e.g., the action probability distribution p, in which case the performance index &phgr; may be a function of the action probability distribution p, or the action probability distribution p may be used as the performance index 0. Alternatively, the intuition module 4815 may be non-existent, or may desire not to modify the probabilistic learning module 4810 depending on the objective of the program 4800.
[0764] The modification of the probabilistic learning module 4810 is generally accomplished similarly to that described with respect to the afore-described probabilistic learning module 110. That is, the functionalities of (1) the probability update module 4820 (e.g., by selecting from a plurality of algorithms used by the probability update module 4820, modifying one or more parameters within an algorithm used by the probability update module 4820, transforming or otherwise modifying the action probability distribution p); (2) the action selection module 4825 (e.g., limiting or expanding selection of the action &agr;x corresponding to a subset of probability values contained within the action probability distribution p); and/or (3) the outcome evaluation module 4830 (e.g., modifying the nature of the outcome value &bgr; or otherwise the algorithms used to determine the outcome values &bgr;, are modified.
[0765] The various different types of learning methodologies previously described herein can be applied to the probabilistic learning module 4810. The operation of the program 4800 is similar to that of the program 2200 described with respect to FIG. 31, with the exception that the program 4800 treats an action pair or trio as an action. Specifically, referring to FIG. 68, the probability update module 4820 initializes the action probability distribution p (step 4850) similarly to that described with respect to step 150 of FIG. 4. The action selection module 4825 then determines if one or more of the user actions &lgr;x1-&lgr;x3 have been selected by the users 4805(1)-(3) from the respective user action sets &lgr;1-&lgr;3 (step 4855). If not, the program 4800 does not select a processor action &agr;x from the processor action set &agr; (step 4860), or alternatively selects a processor action &agr;x, e.g., randomly, notwithstanding that none of the user actions &lgr;x1-&lgr;x3 has been selected (step 4865), and then returns to step 4855 where it again determines if one or more of the user actions &lgr;x1-&lgr;x3 have been selected. If one or more of the user actions &lgr;x1-&lgr;x3 have been performed at step 4855, the action selection module 4825 determines the nature of the selected ones of the user actions &lgr;x1-&lgr;x3.
[0766] Specifically, the action selection module 4825 determines whether any of the selected ones of the user actions &lgr;x1-&lgr;x3 are of the type that should be countered with a processor action &agr;x (step 4870). If so, the action selection module 4825 selects a processor action &agr;x from the processor action set &agr; based on the action probability distribution p (step 4875). The probability values pui within the action probability distribution p will correspond to the user/processor action pairs &agr;uix. Alternatively, an action probability distribution p containing probability values puxix corresponding to user/user action/processor action trios &agr;uxix can be used, or in the case of a single user, probability values pxi corresponding to user action/processor action pairs &agr;xi. After the performance of step 4875, or if the action selection module 4825 determines that none of the selected user actions &lgr;x1-&lgr;x3 is of the type that should be countered with a processor action &agr;x, the action selection module 4825 determines if any of the selected user actions &lgr;x1-&lgr;x3 are of the type that the performance index &phgr; is based on (step 4880).
[0767] If not, the program 4800 returns to step 4855 to determine again whether any of the user actions &lgr;x1-&lgr;x3 have been selected. If so, the outcome evaluation module 4830 quantifies the performance of the previously selected processor action &agr;x relative to the currently selected user actions &lgr;x1-&lgr;x3 by generating outcome values &bgr; (&bgr;uix, &bgr;xix or &bgr;uxix) (step 4885). The intuition module 4815 then updates the performance index &phgr; based on the outcome values &bgr; unless the performance index &phgr; is an instantaneous performance index that is represented by the outcome values &bgr; themselves (step 4890), and modifies the probabilistic learning module 4810 by modifying the functionalities of the probability update module 4820, action selection module 4825, or outcome evaluation module 4830 (step 4895). The probability update module 4820 then, using any of the updating techniques described herein, updates the action probability distribution p based on the generated outcome values ,i(step 4898).
[0768] The program 4800 then returns to step 4855 to determine again whether any of the user actions &lgr;x1-&lgr;x3 have been selected. It should be noted that the order of the steps described in FIG. 68 may vary depending on the specific application of the program 4800.
[0769] Multi-Player Game Program (Single Game Move-Teacher Action Pair)
[0770] Having now generally described the components and functionality of the learning program 4800, we now describe one of its various applications. Referring to FIG. 69, a multiple-player game program 4900 developed in accordance with the present inventions is described in the context of the previously described duck hunting game 2300 (see FIG. 32).
[0771] The game program 4900 generally includes a probabilistic learning module 4910 and an intuition module 4915, which are specifically tailored for the game 2300. The probabilistic learning module 4910 comprises a probability update module 4920, a game move selection module 4925, and an outcome evaluation module 4930 that are similar to the previously described probability update module 2420, game move selection module 2425, and outcome evaluation module 2430, with the exception that the probability update module 4920 updates probability values corresponding to player/game move pairs, rather than single game moves. The game move probability distribution p that the probability update module 4920 generates and updates can be represented by the following equation:
p(k)=[p1,1(k),p1,2(k),p1,3(k) . . . p2,1(k),p2,2(k),p2,3(k) . . .pmn(k)], [46]
[0772] where pui is the game move probability value assigned to a specific player/game move pair &agr;uix; m is the number of players; n is the number of game moves &agr;x within the game move set &agr;, and k is the incremental time at which the game move probability distribution was updated.
[0773] The game program 4900 may employ the following P-type Teacher Action Pair (TAP) SISO equations: 48 p ui ( k + 1 ) = p ui ( k ) + ∑ t , s = 1 , 1 t , s ≠ u , i n , m g ts ( p ( k ) ) ; if α ( k ) = α ui and β ui ( k ) = 1 [ 47 ]
puix(k+1)=puix(k)−guix(p(k)), if &agr;(k)≠&agr;uix and &bgr;uix(k)=1 [48] 49 p ui ( k + 1 ) = p ui ( k ) - ∑ t , s = 1 , 1 t , s ≠ u , i n , m h ts ( p ( k ) ) ; if α ( k ) = α ui and β ui ( k ) = 0 [ 49 ] puix(k+1)=puix(k)+huix(p(k)), if &agr;(k)≠&agr;uix and &bgr;uix(k)=0 [50]
[0774] where puix(k+1) and puix(k), m, and n have been previously defined, guix(p(k)) and huix(p(k)) are respective reward and penalty functions, u is an index for the player, i is an index for the currently selected game move &agr;x, and &bgr;uix(k) is the outcome value based on a selected game move &agr;i relative to an action &lgr;x selected by the player.
[0775] As an example, if there are a total of three players and ten game moves, the game move probability distribution p will have probability values puix corresponding to player/game move pairs &agr;uix, as set forth in Table 22. 21 TABLE 22 Probability Values for Player/Game Move Pairs Given Ten Moves and Three Players &agr;1 &agr;2 &agr;3 &agr;4 &agr;5 &agr;6 &agr;7 &agr;8 &agr;9 &agr;10 P1 p1,1 p1,2 p1,3 p1,4 p1,5 p1,6 p1,7 p1,8 p1,9 p1,10 P2 p2,1 p2,2 p2,3 p2,4 p2,5 p2,6 p2,7 p2,8 p2,9 p2,10 P3 p3,1 p3,2 p3,3 p3,4 p3,5 p3,6 p3,7 p3,8 p3,9 p3,10
[0776] Having now described the structure of the game program 4900, the steps performed by the game program 4900 will be described with reference to FIG. 70. First, the probability update module 4920 initializes the game move probability distribution p and current game move &agr;x (step 5005) similarly to that described in step 405 of FIG. 9. Then, the game move selection module 4925 determines whether one of the player moves &lgr;2x1-&lgr;2x3 has been performed, and specifically whether one of the guns 2325(1)-(3) has been fired (step 5010). If one of the player moves &lgr;2x1-&lgr;2x3 has been performed, the outcome evaluation module 4930 generates the corresponding outcome value &bgr;uix for the performed one of the player moves &lgr;2x1-&lgr;2x3 (step 5015), and the intuition module 4915 then updates the corresponding one of the player scores 2360(1)-(3) and duck scores 2365(1)-(3) based on the outcome value &agr;uix (step 5020), similarly to that described in steps 415 and 420 of FIG. 9. The probability update module 4920 then, using the TAP SISO equations [47]-[50], updates the game move probability distributionp based on the generated outcome value &bgr;uix, (step 5025).
[0777] After step 5025, or if none of the player moves &lgr;2x1-&lgr;2x3 has been performed at step 5010, the game move selection module 4925 determines if any of the player moves &lgr;1x1-&lgr;1x3 have been performed, i.e., guns 2325(1)-(3), have breached the gun detection region 270 (step 5030). If none of the guns 2325(1)-(3) has breached the gun detection region 270, the game move selection module 4925 does not select a game move &agr;x from the game move set &agr; and the duck 2320 remains in the same location (step 5035). Alternatively, the game move &agr;x may be randomly selected, allowing the duck 2320 to dynamically wander. The game program 4900 then returns to step 5010 where it is again determined if any of the player moves &lgr;1x1-&lgr;1x3 has been performed. If any of the guns 2325(1)-(3) have breached the gun detection region 270 at step 5030, the intuition module 4915 modifies the functionality of the game move selection module 4925 based on the performance index &phgr;, and the game move selection module 4925 selects a game move &agr;x from the game move set &agr; in the manner previously described with respect to steps 440-470 of FIG. 9 (step 5040). It should be noted that, rather than use the game move subset selection technique, other afore-described techniques used to dynamically and continuously match the skill level of the players 2315(1)-(3) with the skill level of the game 2300, such as that illustrated in FIG. 10, can be alternatively or optionally be used as well in the game program 3900.
[0778] Although particular embodiments of the present inventions have been shown and described, it will be understood that it is not intended to limit the present inventions to the preferred embodiments, and it will be obvious to those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the present inventions. Thus, the present inventions are intended to cover alternatives, modifications, and equivalents, which may be included within the spirit and scope of the present inventions as defined by the claims. All publications, patents, and patent applications cited herein are hereby incorporated by reference in their entirety for all purposes.
Claims
1. A method of providing learning capability to a processing device having one or more objectives, comprising:
- identifying an action performed by a user;
- selecting one of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- determining an outcome of one or both of said identified user action and said selected processor action;
- updating said action probability distribution using a learning automaton based on said outcome; and
- modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
2. The method of claim 1, wherein said outcome is of said identified user action.
3. The method of claim 1, wherein said outcome is of said selected processor action.
4. The method of claim 1, wherein said outcome is of said selected processor action relative to said identified user action.
5. The method of claim 1, wherein said selected processor action is selected in response to said identified user action.
6. The method of claim 1, further comprising determining a performance index indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said performance index.
7. The method of claim 6, wherein said performance index is updated when an outcome is determined.
8. The method of claim 6, wherein said performance index is derived from an outcome determination.
9. The method of claim 6, wherein said performance index is derived indirectly from an outcome determination.
10. The method of claim 6, wherein said performance index is a function of said action probability distribution.
11. The method of claim 6, wherein said performance index is cumulative.
12. The method of claim 6, wherein said performance index is instantaneous.
13. The method of claim 1, wherein said modification is performed deterministically.
14. The method of claim 1, wherein said modification is performed quasi-deterministically.
15. The method of claim 1, wherein said modification is performed probabilistically.
16. The method of claim 1, wherein said modification is performed using artificial intelligence.
17. The method of claim 1, wherein said modification is performed using an expert system.
18. The method of claim 1, wherein said modification is performed using a neural network.
19. The method of claim 1, wherein said modification is performed using fuzzy logic.
20. The method of claim 1, wherein said modification comprises modifying said action selection.
21. The method of claim 1, wherein said modification comprises modifying said outcome determination.
22. The method of claim 1, wherein said modification comprises modifying said action probability distribution update.
23. The method of claim 1, wherein said modification comprises selecting one of a predetermined plurality of algorithms employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
24. The method of claim 1, wherein said modification comprises modifying a parameter of an algorithm employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
25. The method of claim 1, wherein said outcome can be represented by one of two possible values.
26. The method of claim 25, wherein said two possible values are the integers “zero” and “one.”
27. The method of claim 1, wherein said outcome can be represented by one of a finite range of real numbers.
28. The method of claim 1, wherein said outcome can be represented by one of a range of continuous values.
29. The method of claim 1, wherein said selected processor action is a currently selected processor action relative to said action probability distribution update.
30. The method of claim 1, wherein said selected processor action is a previously selected processor action relative to said action probability distribution update.
31. The method of claim 1, wherein said selected processor action is a subsequently selected processor action relative to said action probability distribution update.
32. The method of claim 1, further comprising initially generating said action probability distribution with equal probability values.
33. The method of claim 1, further comprising initially generating said action probability distribution with unequal probability values.
34. The method of claim 1, wherein said action probability distribution update comprises a linear update.
35. The method of claim 1, wherein said action probability distribution update comprises a nonlinear update.
36. The method of claim 1, wherein said action probability distribution update comprises an absolutely expedient update.
37. The method of claim 1, wherein said action probability distribution update comprises a reward-penalty update.
38. The method of claim 1, wherein said action probability distribution update comprises a reward-inaction update.
39. The method of claim 1, wherein said action probability distribution update comprises a inaction-penalty update.
40. The method of claim 1, wherein said action probability distribution is normalized.
41. The method of claim 1, wherein said selected processor action corresponds to the highest probability value within said action probability distribution.
42. The method of claim 1, wherein said selected processor action corresponds to a pseudo- random selection of a probability value within said action probability distribution.
43. The method of claim 1, wherein said processing device is a computer game, said identified user action is a player move, and said processor actions are game moves.
44. The method of claim 1, wherein said processing device is an educational toy, said identified user action is a child action, and said processor actions are toy actions.
45. The method of claim 1, wherein said processing device is a telephone system, said identified user action is a called phone number, and said processor actions are listed phone numbers.
46. The method of claim 1, wherein said processing device is a television channel control system, said identified user action is a watched television channel, and said processor actions are listed television channels.
47. A processing device having one or more objectives, comprising:
- a probabilistic learning module having a learning automaton configured for learning a plurality of processor actions in response to a plurality of actions performed by a user; and
- an intuition module configured for modifying a functionality of said probabilistic learning module based on said one or more objectives.
48. The processing device of claim 47, wherein said intuition module is further configured for determining a performance index indicative of a performance of said probabilistic learning module relative to said one or more objectives, and for modifying said probabilistic learning module functionality based on said performance index.
49. The processing device of claim 47, wherein said intuition module is deterministic.
50. The processing device of claim 47, wherein said intuition module is quasi-deterministic.
51. The processing device of claim 47, wherein said intuition module is probabilistic.
52. The processing device of claim 47, wherein said intuition module comprises artificial intelligence.
53. The processing device of claim 47, wherein said intuition module comprises an expert system.
54. The processing device of claim 47, wherein said intuition module comprises a neural network.
55. The processing device of claim 47, wherein said intuition module comprises fuzzy logic.
56. The processing device of claim 47, wherein said probabilistic learning module comprises:
- an action selection module configured for selecting one of a plurality of processor actions, said action selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- an outcome evaluation module configured for determining an outcome of one or both of said identified user action and said selected processor action; and
- a probability update module configured for updating said action probability distribution based on said outcome.
57. The processing device of claim 56, wherein said outcome is of said identified user action.
58. The processing device of claim 56, wherein said outcome is of said selected processor action.
59. The processing device of claim 56, wherein said outcome is of said selected processor action relative to said identified user action.
60. The processing device of claim 56, wherein said intuition module is configured for modifying a functionality of said action selection module based on said one or more objectives.
61. The processing device of claim 56, wherein said intuition module is configured for modifying a functionality of said outcome evaluation module based on said one or more objectives.
62. The processing device of claim 56, wherein said intuition module is configured for modifying a functionality of said probability update module based on said one or more objectives.
63. The processing device of claim 47, wherein said intuition module is configured for selecting one of a predetermined plurality of algorithms employed by said learning module.
64. The processing device of claim 47, wherein said intuition module is configured for modifying a parameter of an algorithm employed by said learning module.
65. A method of providing learning capability to a computer game having an objective of matching a skill level of said computer game with a skill level of a game player, comprising:
- identifying a move performed by said game player;
- selecting one of a plurality of game moves based on a game move probability distribution comprising a plurality of probability values corresponding to said plurality of game moves;
- determining an outcome of said selected game move relative to said identified player move;
- updating said game move probability distribution based on said outcome; and
- modifying one or more of said game move selection, said outcome determination, and said game move probability distribution update based on said objective.
66. The method of claim 65, wherein said selected game move is selected in response to said identified player move.
67. The method of claim 65, further comprising determining a performance index indicative of a performance of said computer game relative to said objective, wherein said modification is based on said performance index.
68. The method of claim 67, wherein said performance index comprises a relative score value between said game player and said computer game.
69. The method of claim 65, wherein said performance index is updated when an outcome is determined.
70. The method of claim 67, wherein said performance index is derived from an outcome determination.
71. The method of claim 67, wherein said performance index is derived indirectly from an outcome determination.
72. The method of claim 67, wherein said performance index is a function of said game move probability distribution.
73. The method of claim 67, wherein said performance index is cumulative.
74. The method of claim 67, wherein said performance index is instantaneous.
75. The method of claim 65, wherein said modification comprises modifying said game move selection.
76. The method of claim 75, wherein said plurality of game moves are organized into a plurality of game move subsets, said selected game move is selected from one of said plurality of game move subsets, and said subsequent game move selection comprises selecting another of said plurality of game move subsets.
77. The method of claim 76, wherein said game move selection comprises selecting another game move from said another of said plurality of game move subsets in response to another player move.
78. The method of claim 65, wherein said modification comprises modifying said outcome determination.
79. The method of claim 65, wherein said modification comprises modifying said game move probability distribution update.
80. The method of claim 65, wherein said modification comprises selecting one of a predetermined plurality of algorithms employed by said one or more of said performed game move selection, said outcome determination, and said game move probability distribution update.
81. The method of claim 65, wherein said modification comprises modifying a parameter of an algorithm employed by said one or more of said game move selection, said outcome determination, and said game move probability distribution update.
82. The method of claim 65, wherein said outcome can be represented by one of two possible values.
83. The method of claim 82, wherein said two possible values are the integers “zero” and “one.”
84. The method of claim 65, wherein said outcome can be represented by one of a finite range of real numbers.
85. The method of claim 65, wherein said outcome can be represented by one of a range of continuous values.
86. The method of claim 65, wherein said selected game move is a currently selected game move.
87. The method of claim 65, wherein said selected game move is a previously selected game move.
88. The method of claim 65, wherein said selected game move is a subsequently selected game move.
89. The method of claim 65, wherein said outcome is determined by performing a collision technique on said identified player move and said selected game move.
90. The method of claim 65, further comprising initially generating said game move probability distribution with equal probability values.
91. The method of claim 65, further comprising initially generating said game move probability distribution with unequal probability values.
92. The method of claim 65, wherein said game move probability distribution update comprises a linear update.
93. The method of claim 65, wherein said game move probability distribution update comprises a nonlinear update.
94. The method of claim 65, wherein said game move probability distribution update comprises an absolutely expedient update.
95. The method of claim 65, wherein said game move probability distribution update comprises a reward-penalty update.
96. The method of claim 65, wherein said game move probability distribution update comprises a reward-inaction update.
97. The method of claim 65, wherein said game move probability distribution update comprises a inaction-penalty update.
98. The method of claim 65, wherein said game move probability distribution is normalized.
99. The method of claim 65, wherein said selected game move corresponds to the highest probability value within said game move probability distribution.
100. The method of claim 65, wherein said selected game move corresponds to a pseudo- random selection of a probability value within said game move probability distribution.
101. The method of claim 65, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
102. The method of claim 101, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
103. The method of claim 101, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
104. The method of claim 101, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
105. The method of claim 101, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
106. The method of claim 65, wherein said game move probability distribution is updated using a learning automaton.
107. A computer game having an objective of matching a skill level of said computer game with a skill level of a game player, comprising:
- a probabilistic learning module configured for learning a plurality of game moves in response to a plurality of moves performed by a game player; and
- an intuition module configured for modifying a functionality of said probabilistic learning module based on said objective.
108. The computer game of claim 107, wherein said intuition module is further configured for determining a performance index indicative of a performance of said probabilistic learning module relative to said objective, and for modifying said probabilistic learning module functionality based on said performance index.
109. The computer game of claim 108, wherein said performance index comprises a relative score value between said game player and said computer game.
110. The computer game of claim 107, wherein said probabilistic learning module comprises:
- a game move selection module configured for selecting one of a plurality of game moves, said game move selection being based on a game move probability distribution comprising a plurality of probability values corresponding to said plurality of game moves;
- an outcome evaluation module configured for determining an outcome of said selected game move relative to said identified player move; and
- a probability update module configured for updating said game move probability distribution based on said outcome.
111. The computer game of claim 110, wherein said intuition module is configured for modifying a functionality of said game move selection module based on said objective.
112. The computer game of claim 1 10, wherein said intuition module is configured for modifying a functionality of said outcome evaluation module based on said objective.
113. The computer game of claim 1 10, wherein said intuition module is configured for modifying a functionality of said probability update module based on said objective.
114. The computer game of claim 110, wherein said intuition module is configured for selecting one of a predetermined plurality of algorithms employed by said learning module.
115. The computer game of claim 1 10, wherein said intuition module is configured for modifying a parameter of an algorithm employed by said learning module.
116. The computer game of claim 1 10, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
117. The computer game of claim 116, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
118. The computer game of claim 116, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
119. The computer game of claim 116, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
120. The computer game of claim 116, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
121. The computer game of claim 107, wherein said probabilistic learning module comprises a learning automaton.
122. A method of providing learning capability to a processing device, comprising:
- generating an action probability distribution comprising a plurality of probability values organized among a plurality of action subsets, said plurality of probability values corresponding to a plurality of processor actions;
- selecting one of said plurality of action subsets; and
- selecting one of said plurality of processor actions from said selected action subset.
123. The method of claim 122, further comprising:
- identifying an action performed by a user, determining an outcome of said selected processor action relative to said identified user action; and
- updating said action probability distribution based on said outcome.
124. The method of claim 123, wherein said selected processor action is selected in response to said identified user action.
125. The method of claim 122, wherein said processing device has one or more objectives, the method further comprising determining a performance index indicative of a performance of said processing device relative to one or more objectives of said processing device, wherein said action subset selection is based on said performance index.
126. The method of claim 122, wherein said selected action subset is selected deterministically.
127. The method of claim 122, wherein said selected action subset is selected quasi-deterministically.
128. The method of claim 122, wherein said selected action subset is selected probabilistically.
129. The method of claim 122, wherein said selected processor action is pseudo-randomly selected from said selected action subset.
130. The method of claim 122, wherein said selected action subset corresponds to a series of probability values within said action probability distribution.
131. The method of claim 122, wherein said selected action subset corresponds to the highest probability values within said action probability distribution.
132. The method of claim 122, wherein said selected action subset corresponds to the lowest probability values within said action probability distribution.
133. The method of claim 122, wherein said selected action subset corresponds to the middlemost probability values within said action probability distribution.
134. The method of claim 122, wherein said selected action subset corresponds to probability values having an average relative to a threshold value.
135. The method of claim 134, wherein said threshold value is a median probability value within said action probability distribution.
136. The method of claim 134, wherein said threshold value is dynamically adjusted.
137. The method of claim 134, wherein said selected action subset corresponds to probability values having an average greater than said threshold value.
138. The method of claim 134, wherein said selected action subset corresponds to probability values having an average less than said threshold value.
139. The method of claim 134, wherein said selected action subset corresponds to probability values having an average substantially equal to said threshold value.
140. The method of claim 122, wherein said action probability distribution is updated using a learning automaton.
141. A method of providing learning capability to a computer game, comprising:
- generating a game move probability distribution comprising a plurality of probability values organized among a plurality of game move subsets, said plurality of probability values corresponding to a plurality of game moves;
- selecting one of said plurality of game move subsets; and
- selecting one of said plurality of game moves from said selected game move subset.
142. The method of claim 141, wherein said selected game move is selected in response to said identified player move.
143. The method of claim 141, further comprising:
- identifying an move performed by a game player;
- determining an outcome of said selected game move relative to said identified player move; and
- updating said game move probability distribution based on said outcome.
144. The method of claim 143, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
145. The method of claim 144, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
146. The method of claim 144, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
147. The method of claim 144, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
148. The method of claim 144, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
149. The method of claim 141, wherein said selected game move subset is selected deterministically.
150. The method of claim 141, wherein said selected game move subset is selected quasi-deterministically.
151. The method of claim 141, wherein said selected game move subset is selected probabilistically.
152. The method of claim 141, wherein said selected game move is pseudo-randomly selected from said selected game move subset.
153. The method of claim 141, wherein said selected game move subset corresponds to a series of probability values within said game move probability distribution.
154. The method of claim 141, wherein said selected game move subset corresponds to the highest probability values within said game move probability distribution.
155. The method of claim 141, wherein said selected game move subset corresponds to the lowest probability values within said game move probability distribution.
156. The method of claim 141, wherein said selected game move subset corresponds to the middlemost probability values within said game move probability distribution.
157. The method of claim 141, wherein said selected game move subset corresponds to probability values having an average relative to a threshold level.
158. The method of claim 157, wherein said threshold level is a median probability value within said game move probability distribution.
159. The method of claim 157, wherein said threshold level is dynamically adjusted.
160. The method of claim 157, wherein said selected game move subset corresponds to probability values having an average greater than said threshold level.
161. The method of claim 157, wherein said selected game move subset corresponds to probability values having an average less than said threshold level.
162. The method of claim 157, wherein said selected game move subset corresponds to probability values having an average substantially equal to said threshold level.
163. The method of claim 141, wherein said selected game move subset is selected based on a skill level of a game player relative to a skill level of said computer game.
164. The method of claim 163, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
165. The method of claim 163, wherein said game move subset is selected to correspond to the highest probability values within said game move probability distribution if said relative skill level is greater than a threshold level.
166. The method of claim 163, wherein said game move subset is selected to correspond to the lowest probability values within said game move probability distribution if said relative skill level is less than a threshold level.
167. The method of claim 163, wherein said game move subset is selected to correspond to the middlemost probability values within said game move probability distribution if said relative skill level is within a threshold range.
168. The method of claim 163, wherein said game move subset is selected to correspond to probability values having an average relative to a threshold level.
169. The method of claim 168, wherein said threshold level is a median probability value within said game move probability distribution.
170. The method of claim 168, wherein said threshold level is dynamically adjusted based on said relative skill level.
171. The method of claim 168, wherein said game move subset is selected to correspond to probability values having an average greater than said threshold level if said relative skill level value is greater than a relative skill threshold level.
172. The method of claim 168, wherein said game move subset is selected to correspond to probability values having an average less than said threshold level if said relative skill level value is less than a relative skill threshold level.
173. The method of claim 168, wherein said game move subset is selected to correspond to probability values having an average substantially equal to said threshold level if said relative skill level value is within a relative skill threshold range.
174. The method of claim 141, wherein said game move probability distribution is updated using a learning automaton.
175. A method of providing learning capability to a processing device, comprising:
- generating an action probability distribution using one or more learning algorithms, said action probability distribution comprising a plurality of probability values corresponding to a plurality of processor actions;
- modifying said one or more learning algorithms; and
- updating said action probability distribution using said modified one or more learning algorithms.
176. The method of claim 175, further comprising:
- identifying an action performed by a user;
- selecting one of said plurality of processor actions; and
- determining an outcome of one or both of said identified user action and said selected processor action, wherein said action probability distribution update is based on said outcome.
177. The method of claim 176, wherein said outcome is of said identified user action.
178. The method of claim 176, wherein said outcome is of said selected processor action.
179. The method of claim 176, wherein said outcome is of said selected processor action relative to said identified user action.
180. The method of claim 176, wherein said selected processor action is selected in response to said identified user action.
181. The method of claim 175, wherein said processing device has one or more objectives, the method further comprising determining a performance index indicative of a performance of said processing device relative to said one or more objectives, wherein said algorithm modification is based on said performance index.
182. The method of claim 175, wherein said one or more learning algorithms are modified deterministically.
183. The method of claim 175, wherein said one or more learning algorithms are modified quasi-deterministically.
184. The method of claim 175, wherein said one or more learning algorithms are modified probabilistically.
185. The method of claim 175, wherein said one or more algorithms comprises one or more parameters, and said algorithm modification comprises modifying said one or more parameters.
186. The method of claim 175, wherein said one or more parameters are modified in accordance with a function.
187. The method of claim 185, wherein said one or more parameters comprises a reward parameter.
188. The method of claim 185, wherein said one or more parameters comprises a penalty parameter.
189. The method of claim 185, wherein said one or more parameters comprises one or more of a reward parameter and penalty parameter.
190. The method of claim 189, wherein said one or more of a reward parameter and penalty parameter are increased.
191. The method of claim 189, wherein said one or more of a reward parameter and penalty parameter are decreased.
192. The method of claim 189, wherein said one or more of a reward parameter and penalty parameter are modified to a negative number.
193. The method of claim 185, wherein said one or more parameters comprises a reward parameter and a penalty parameter.
194. The method of claim 193, wherein said reward parameter and said penalty parameter are both increased.
195. The method of claim 193, wherein said reward parameter and said penalty parameter are both decreased.
196. The method of claim 193, wherein said reward parameter and said penalty parameter are modified to a negative number.
197. The method of claim 175, wherein said one or more algorithms is linear.
198. The method of claim 175, wherein said action probability distribution is updated using a learning automaton.
199. A method of providing learning capability to a computer game, comprising:
- generating a game move probability distribution using one or more learning algorithms, said game move probability distribution comprising a plurality of probability values corresponding to a plurality of game moves;
- modifying said one or more learning algorithms; and
- updating said game move probability distribution using said modified one or more learning algorithms.
200. The method of claim 199, further comprising:
- identifying an move performed by a game player;
- selecting one of said plurality of game moves; and
- determining an outcome of said selected game move relative to said identified player move, wherein said action probability distribution update is based on said outcome.
201. The method of claim 200, wherein said selected game move is selected in response to said identified player move.
202. The method of claim 199, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
203. The method of claim 202, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
204. The method of claim 202, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
205. The method of claim 202, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
206. The method of claim 202, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
207. The method of claim 199, wherein said one or more learning algorithms are modified deterministically.
208. The method of claim 199, wherein said one or more learning algorithms are modified quasi-deterministically.
209. The method of claim 199, wherein said one or more learning algorithms are modified probabilistically.
210. The method of claim 199, wherein said one or more algorithms comprises one or more parameters, and said algorithm modification comprises modifying said one or more parameters.
211. The method of claim 210, wherein said one or more parameters are modified in accordance with a function.
212. The method of claim 210, wherein said one or more parameters comprises a reward parameter.
213. The method of claim 210, wherein said one or more parameters comprises a penalty parameter.
214. The method of claim 210, wherein said one or more parameters comprises one or more of a reward parameter and penalty parameter.
215. The method of claim 214, wherein said one or more of a reward parameter and penalty parameter are increased.
216. The method of claim 214, wherein said one or more of a reward parameter and penalty parameter are decreased.
217. The method of claim 214, wherein said one or more of a reward parameter and penalty parameter are modified to a negative number.
218. The method of claim 210, wherein said one or more parameters comprises a reward parameter and a penalty parameter.
219. The method of claim 218, wherein said reward parameter and said penalty parameter are both increased.
220. The method of claim 218, wherein said reward parameter and said penalty parameter are both decreased.
221. The method of claim 218, wherein said reward parameter and said penalty parameter are modified to a negative number.
222. The method of claim 210, wherein said modified one or more algorithms is modified based on a skill level of a game player relative to a skill level of said computer game.
223. The method of claim 210, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
224. The method of claim 210, wherein said one or more algorithms comprises one or more of a reward parameter and a penalty parameter, and said algorithm modification comprises modifying said one or more of a reward parameter and a penalty parameter based on a skill level of game player relative to a skill level of said computer game.
225. The method of claim 224, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
226. The method of claim 224, wherein said one or more of a reward parameter and a penalty parameter is increased if said relative skill level is greater than a threshold level.
227. The method of claim 224, wherein said one or more of a reward parameter and a penalty parameter is decreased if said relative skill level is less than a threshold level.
228. The method of claim 224, wherein said one or more of a reward parameter and a penalty parameter is modified to be a negative number if said relative skill level is less than a threshold level.
229. The method of claim 199, wherein said one or more algorithms is linear.
230. The method of claim 199, wherein said one or more algorithms comprises a reward parameter and a penalty parameter, and said algorithm modification comprises modifying both of said reward parameter and said penalty parameter based on a skill level of game player relative to a skill level of said computer game.
231. The method of claim 230, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
232. The method of claim 230, wherein both of said reward parameter and said penalty parameter are increased if said relative skill level is greater than a threshold level.
233. The method of claim 230, wherein both of said reward parameter and said penalty parameter are decreased if said relative skill level is less than a threshold level.
234. The method of claim 230, wherein both of said reward parameter and said penalty parameter are modified to be a negative number if said relative skill level is less than a threshold level.
235. The method of claim 230, wherein said one or more algorithms is linear.
236. The method of claim 199, wherein said game move probability distribution is updated using a learning automaton.
237. A method of matching a skill level of game player with a skill level of a computer game, comprising:
- identifying a move performed by said game player;
- selecting one of a plurality of game moves based on a game move probability distribution comprising a plurality of probability values corresponding to said plurality of game moves;
- determining if said selected game move is successful;
- determining a current skill level of said game player relative to a current skill level of said computer game; and
- updating said game move probability distribution using a reward algorithm if said selected game move is successful and said relative skill level is relatively high, or if said selected game move is unsuccessful and said relative skill level is relatively low.
238. The method of claim 237, wherein said selected game move is selected in response to said identified player move.
239. The method of claim 237, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
240. The method of claim 237, wherein said relative skill level is determined to be relatively high if greater than a first threshold value, and relatively low if lower than a second threshold value.
241. The method of claim 237, wherein said reward algorithm is linear.
242. The method of claim 237, further comprising modifying said reward algorithm based on said successful game move determination.
243. The method of claim 237, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
244. The method of claim 243, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
245. The method of claim 243, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
246. The method of claim 243, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
247. The method of claim 243, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
248. The method of claim 237, wherein said game move probability distribution is updated using a learning automaton.
249. A method of matching a skill level of game player with a skill level of a computer game, comprising:
- identifying a move performed by said game player;
- selecting one of a plurality of game moves based on a game move probability distribution comprising a plurality of probability values corresponding to said plurality of game moves;
- determining if said selected game move is successful;
- determining a current skill level of said game player relative to a current skill level of said computer game; and
- updating said game move probability distribution using a penalty algorithm if said selected game move is unsuccessful and said relative skill level is relatively high, or if said selected game move is successful and said relative skill level is relatively low.
250. The method of claim 249, wherein said selected game move is selected in response to said identified player move.
251. The method of claim 249, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
252. The method of claim 249, wherein said relative skill level is determined to be relatively high if greater than a first threshold value, and relatively low if lower than a second threshold value.
253. The method of claim 249, wherein said penalty algorithm is linear.
254. The method of claim 249, further comprising modifying said penalty algorithm based on said successful game move determination.
255. The method of claim 249, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
256. The method of claim 255, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
257. The method of claim 255, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
258. The method of claim 255, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
259. The method of claim 255, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
260. The method of claim 249, wherein said game move probability distribution is updated using a learning automaton.
261. A method of matching a skill level of game player with a skill level of a computer game, comprising:
- identifying a move performed by said game player;
- selecting one of a plurality of game moves based on a game move probability distribution comprising a plurality of probability values corresponding to said plurality of game moves;
- determining if said selected game move is successful;
- determining a current skill level of said game player relative to a current skill level of said computer game;
- updating said game move probability distribution using a reward algorithm if said selected game move is successful and said relative skill level is relatively high, or if said selected game move is unsuccessful and said relative skill level is relatively low; and
- updating said game move probability distribution using a penalty algorithm if said selected game move is unsuccessful and said relative skill level is relatively high, or if said selected game move is successful and said relative skill level is relatively low.
262. The method of claim 261, wherein said selected game move is selected in response to said identified player move.
263. The method of claim 261, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
264. The method of claim 261, wherein said relative skill level is determined to be relatively high if greater than a first threshold value, and relatively low if lower than a second threshold value.
265. The method of claim 261, wherein said reward algorithm and said penalty algorithm are linear.
266. The method of claim 261, further comprising modifying said reward algorithm and said penalty algorithm based on said successful game move determination.
267. The method of claim 261, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
268. The method of claim 267, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
269. The method of claim 267, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
270. The method of claim 267, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
271. The method of claim 267, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
272. The method of claim 261, wherein said game move probability distribution is updated using a learning automaton.
273. A method of matching a skill level of game player with a skill level of a computer game, comprising:
- identifying a move performed by said game player;
- selecting one of a plurality of game moves based on a game move probability distribution comprising a plurality of probability values corresponding to said plurality of game moves;
- determining if said selected game move is successful;
- determining a current skill level of said game player relative to a current skill level of said computer game;
- generating a successful outcome if said selected game move is successful and said relative skill level is relatively high, or if said selected game move is unsuccessful and said relative skill level is relatively low; and
- updating said game move probability distribution based on said successful outcome.
274. The method of claim 273, wherein said selected game move is selected in response to said identified player move.
275. The method of claim 273, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
276. The method of claim 273, wherein said relative skill level is determined to be relatively high if greater than a first threshold value, and relatively low if lower than a second threshold value.
277. The method of claim 273, wherein said successful outcome equals the value “1.”
278. The method of claim 273, wherein said successful outcome equals the value “0.”
279. The method of claim 273, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
280. The method of claim 279, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
281. The method of claim 279, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
282. The method of claim 279, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
283. The method of claim 279, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
284. The method of claim 273, wherein said game move probability distribution is updated using a learning automaton.
285. A method of matching a skill level of game player with a skill level of a computer game, comprising:
- identifying a move performed by said game player;
- selecting one of a plurality of game moves based on a game move probability distribution comprising a plurality of probability values corresponding to said plurality of game moves;
- determining if said selected game move is successful;
- determining a current skill level of said game player relative to a current skill level of said computer game;
- generating an unsuccessful outcome if said selected game move is unsuccessful and said relative skill level is relatively high, or if said selected game move is successful and said relative skill level is relatively low; and
- updating said game move probability distribution based on said unsuccessful outcome.
286. The method of claim 285, wherein said selected game move is selected in response to said identified player move.
287. The method of claim 285, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
288. The method of claim 285, wherein said relative skill level is determined to be relatively high if greater than a first threshold value, and relatively low if lower than a second threshold value.
289. The method of claim 285, wherein said unsuccessful outcome equals the value “1.”
290. The method of claim 285, wherein said unsuccessful outcome equals the value “0.”
291. The method of claim 285, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
292. The method of claim 291, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
293. The method of claim 291, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
294. The method of claim 291, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
295. The method of claim 291, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
296. The method of claim 285, wherein said game move probability distribution is updated using a learning automaton.
297. A method of matching a skill level of game player with a skill level of a computer game, comprising:
- identifying a move performed by said game player;
- selecting one of a plurality of game moves based on a game move probability distribution comprising a plurality of probability values corresponding to said plurality of game moves;
- determining if said selected game move is successful;
- determining a current skill level of said game player relative to a current skill level of said computer game;
- generating a successful outcome if said selected game move is successful and said relative skill level is relatively high, or if said selected game move is successful and said relative skill level is relatively low;
- generating an unsuccessful outcome if said selected game move is unsuccessful and said relative skill level is relatively high, or if said selected game move is successful and said relative skill level is relatively low; and
- updating said game move probability distribution based on said successful outcome and said unsuccessful outcome.
298. The method of claim 297, wherein said selected game move is selected in response to said identified player move.
299. The method of claim 297, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
300. The method of claim 297, wherein said relative skill level is determined to be relatively high if greater than a first threshold value, and relatively low if lower than a second threshold value.
301. The method of claim 297, wherein said successful outcome equals the value “1”, and said unsuccessful outcome equal the value “0.”
302. The method of claim 297, wherein said successful outcome equals the value “0,” and said unsuccessful outcome equal the value “1.”
303. The method of claim 297, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
304. The method of claim 303, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
305. The method of claim 303, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
306. The method of claim 303, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
307. The method of claim 303, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
308. The method of claim 297, wherein said game move probability distribution is updated using a learning automaton.
309. A method of providing learning capability to a processing device, comprising:
- generating an action probability distribution comprising a plurality of probability values corresponding to a plurality of processor actions; and
- transforming said action probability distribution.
310. The method of claim 309, further comprising:
- identifying an action performed by a user;
- selecting one of said plurality of processor actions;
- determining an outcome of said selected processor action relative to said identified user action; and
- updating said action probability distribution prior to said action probability distribution transformation, said action probability distribution update being based on said outcome.
311. The method of claim 310, wherein said selected user action is selected in response to said processor action.
312. The method of claim 309, wherein said processing device has one or more objectives, the method further comprising determining a performance index indicative of a performance of said processing device relative to said one or more objectives, wherein said action probability distribution transformation is based on said performance index.
313. The method of claim 309, wherein said transformation is performed deterministically.
314. The method of claim 309, wherein said transformation is performed modified quasi-deterministically.
315. The method of claim 309, wherein said transformation is performed probabilistically.
316. The method of claim 309, wherein said action probability distribution transformation comprises assigning a value to one or more of said plurality of probability values.
317. The method of claim 309, wherein said action probability distribution transformation comprises switching a higher probability value and a lower probability value.
318. The method of claim 309, wherein said action probability distribution transformation comprises switching a set of highest probability values and a set lowest probability values.
319. The method of claim 309, wherein said action probability distribution is updated using a learning automaton.
320. A method of providing learning capability to a computer game, comprising:
- generating a game move probability distribution comprising a plurality of probability values corresponding to a plurality of game moves; and
- transforming said game move probability distribution.
321. The method of claim 320, further comprising:
- identifying a move performed by a game player;
- selecting one of said plurality of game moves;
- determining an outcome of said selected game move relative to said identified player move; and
- updating said game move probability distribution prior to said game move probability distribution transformation, said game move probability distribution update being based on said outcome.
322. The method of claim 321, wherein said selected game move is selected in response to said identified player move.
323. The method of claim 320, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
324. The method of claim 323, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
325. The method of claim 323, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
326. The method of claim 323, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
327. The method of claim 323, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
328. The method of claim 320, wherein said transformation is performed deterministically.
329. The method of claim 320, wherein said transformation is performed modified quasi- deterministically.
330. The method of claim 320, wherein said transformation is performed probabilistically.
331. The method of claim 320, wherein said game move probability distribution transformation comprises assigning a value to one or more of said plurality of probability values.
332. The method of claim 320, wherein said game move probability distribution transformation comprises switching a higher probability value and a lower probability value.
333. The method of claim 320, wherein said game move probability distribution transformation comprises switching a set of highest probability values and a set lowest probability values.
334. The method of claim 320, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
335. The method of claim 320, wherein said game move probability distribution is transformed based on a skill level of a game player relative to a skill level of said computer game.
336. The method of claim 335, wherein said game move probability distribution transformation comprises switching a higher probability value and a lower probability value if said relative skill level is greater than a threshold level.
337. The method of claim 335, wherein said game move probability distribution transformation comprises switching a set of highest probability values and a set of lowest probability values if said relative skill level is greater than a threshold level.
338. The method of claim 335, wherein said game move probability distribution transformation comprises switching a higher probability value and a lower probability value if said relative skill level is less than a threshold level.
339. The method of claim 335, wherein said game move probability distribution transformation comprises switching a set of highest probability values and a set of lowest probability values if said relative skill level is less than a threshold level.
340. The method of claim 320, wherein said game move probability distribution is updated using a learning automaton.
341. A method of providing learning capability to a processing device, comprising:
- generating an action probability distribution comprising a plurality of probability values corresponding to a plurality of processor actions; and
- limiting one or more of said plurality of probability values.
342. The method of claim 341, further comprising:
- identifying an action performed by a user;
- selecting one of said plurality of processor actions;
- determining an outcome of one or more said identified user action and said selected processor action; and
- updating said action probability distribution based on said outcome.
343. The method of claim 342, wherein said outcome is of said identified user action.
344. The method of claim 342, wherein said outcome is of said selected processor action.
345. The method of claim 342, wherein said outcome is of said selected processor action relative to said identified user action.
346. The method of claim 342, wherein said selected user action is selected in response to said processor action.
347. The method of claim 341, wherein said processing device has one or more objectives, the method further comprising determining a performance index indicative of a performance of said processing device relative to said one or more objectives, wherein said probability value limitation is based on said performance index.
348. The method of claim 341, wherein said one or more probability values are limited to a high value.
349. The method of claim 341, wherein said one or more probability values are limited to a low value.
350. The method of claim 341, wherein said plurality of probability values is limited.
351. The method of claim 341, wherein said action probability distribution is updated using a learning automaton.
352. A method of providing learning capability to a computer game, comprising:
- generating a game move probability distribution comprising a plurality of probability values corresponding to a plurality of game moves; and
- limiting one or more of said plurality of probability values.
353. The method of claim 352, further comprising:
- identifying a move performed by a game player;
- selecting one of said plurality of game moves;
- determining an outcome of said selected game move relative to said identified player move; and
- updating said game move probability distribution based on said outcome.
354. The method of claim 353, wherein said selected game move is selected in response to said identified player move.
355. The method of claim 353, wherein said plurality of game moves is performed by a game-manipulated object, and said identified player move is performed by a user-manipulated object.
356. The method of claim 353, wherein said plurality of game moves comprises discrete movements of said game-manipulated object.
357. The method of claim 353, wherein said plurality of game moves comprises a plurality of delays related to a movement of said game-manipulated object.
358. The method of claim 353, wherein said identified player move comprises a simulated shot taken by said user-manipulated object.
359. The method of claim 353, wherein said game-manipulated object and said user-manipulated object are visual to said game player.
360. The method of claim 352, wherein said one or more probability values are limited to a high value.
361. The method of claim 352, wherein said one or more probability values are limited to a low value.
362. The method of claim 352, wherein said plurality of probability values is limited.
363. The method of claim 352, wherein said one or more probability values is limited based on a skill level of a game player relative to a skill level of said computer game.
364. The method of claim 363, wherein said relative skill level is obtained from a difference between a game player score and a computer game score.
365. The method of claim 352, wherein said game move probability distribution is updated using a learning automaton.
366. A method of providing learning capability to a processing device, comprising:
- identifying an action performed by a user;
- selecting one of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- determining an outcome of one or both of said identified user action and said selected processor action;
- updating said action probability distribution based on said outcome; and
- wherein said action probability distribution is prevented from substantially converging to a single probability value.
367. The method of claim 366, wherein said outcome is of said identified user action.
368. The method of claim 366, wherein said outcome is of said selected processor action.
369. The method of claim 366, wherein said outcome is of said selected processor action relative to said identified user action.
370. The method of claim 366, wherein said selected processor action is selected in response to said identified user action.
371. The method of claim 366, wherein said outcome can be represented by one of two possible values.
372. The method of claim 371, wherein said two possible values are the integers “zero” and “one.”
373. The method of claim 366, wherein said outcome can be represented by one of a finite range of real numbers.
374. The method of claim 366, wherein said outcome can be represented by one of a range of continuous values.
375. The method of claim 366, wherein said selected processor action is a currently selected processor action relative to said action probability distribution update.
376. The method of claim 366, wherein said selected processor action is a previously selected processor action relative to said action probability distribution update.
377. The method of claim 366, wherein said selected processor action is a subsequently selected processor action relative to said action probability distribution update.
378. The method of claim 366, further comprising initially generating said action probability distribution with equal probability values.
379. The method of claim 366, further comprising initially generating said action probability distribution with unequal probability values.
380. The method of claim 366, wherein said action probability distribution update comprises a linear update.
381. The method of claim 366, wherein said action probability distribution update comprises a nonlinear update.
382. The method of claim 366, wherein said action probability distribution update comprises an absolutely expedient update.
383. The method of claim 366, wherein said action probability distribution update comprises a reward-penalty update.
384. The method of claim 366, wherein said action probability distribution update comprises a reward-inaction update.
385. The method of claim 366, wherein said action probability distribution update comprises a inaction-penalty update.
386. The method of claim 366, wherein said action probability distribution is normalized.
387. The method of claim 366, wherein said selected processor action corresponds to the highest probability value within said action probability distribution.
388. The method of claim 366, wherein said selected processor action corresponds to a pseudo-random selection of a probability value within said action probability distribution.
389. The method of claim 366, wherein said processing device is a computer game, said identified user action is a player move, and said processor actions are game moves.
390. The method of claim 366, wherein said processing device is an educational toy, said identified user action is a child action, and said processor actions are toy actions.
391. The method of claim 366, wherein said processing device is a telephone system, said identified user action is a called phone number, and said processor actions are listed phone numbers.
392. The method of claim 366, wherein said processing device is a television channel control system, said identified user action is a watched television channel, and said processor actions are listed television channels.
393. The method of claim 366, wherein said action probability distribution is updated using a learning automaton.
394. A processing device, comprising:
- a probabilistic learning module configured for learning a plurality of processor actions in response to a plurality of actions performed by a user; and
- an intuition module configured for preventing said probabilistic learning module from substantially converging to a single processor action.
395. The processing device of claim 394, wherein said intuition module is deterministic.
396. The processing device of claim 394, wherein said intuition module is quasi-deterministic.
397. The processing device of claim 394, wherein said intuition module is probabilistic.
398. The processing device of claim 394, wherein said intuition module comprises artificial intelligence.
399. The processing device of claim 394, wherein said intuition module comprises an expert system.
400. The processing device of claim 394, wherein said intuition module comprises a neural network.
401. The processing device of claim 394, wherein said intuition module comprises fuzzy logic.
402. The processing device of claim 394, wherein said probabilistic learning module comprises:
- an action selection module configured for selecting one of a plurality of processor actions, said action selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- an outcome evaluation module configured for determining an outcome of one or both of said identified user action and said selected processor action; and
- a probability update module configured for updating said action probability distribution based on said outcome.
403. The processing device of claim 402, wherein said outcome is of said identified user action.
404. The processing device of claim 402, wherein said outcome is of said selected processor action.
405. The processing device of claim 402, wherein said outcome is of said selected processor action relative to said identified user action.
406. The processing device of claim 394, wherein said probabilistic learning module comprises a learning automaton.
407. A method of providing learning capability to a processing device having a function independent of determining an optimum action, comprising:
- identifying an action performed by a user;
- selecting one of a plurality of processor actions, said action selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions, wherein said selected processor action affects said processing device function;
- determining an outcome of said selected processor action relative to said identified user action; and
- updating said action probability distribution based on said outcome.
408. The method of claim 407, wherein said selected processor action is selected in response to said identified user action.
407. The method of claim 407, wherein said outcome can be represented by one of two possible values.
408. The method of claim 409, wherein said two possible values are the integers “zero” and “one.”
409. The method of claim 407, wherein said outcome can be represented by one of a finite range of real numbers.
409. The method of claim 407, wherein said outcome can be represented by one of a range of continuous values.
410. The method of claim 407, wherein said selected processor action is a currently selected processor action relative to said action probability distribution update.
411. The method of claim 407, wherein said selected processor action is a previously selected processor action relative to said action probability distribution update.
412. The method of claim 407, wherein said selected processor action is a subsequently selected processor action relative to said action probability distribution update.
413. The method of claim 407, further comprising initially generating said action probability distribution with equal probability values.
414. The method of claim 407, further comprising initially generating said action probability distribution with unequal probability values.
415. The method of claim 407, wherein said action probability distribution update comprises a linear update.
416. The method of claim 407, wherein said action probability distribution update comprises a nonlinear update.
417. The method of claim 407, wherein said action probability distribution update comprises an absolutely expedient update.
418. The method of claim 407, wherein said action probability distribution update comprises a reward-penalty update.
419. The method of claim 407, wherein said action probability distribution update comprises a reward-inaction update.
420. The method of claim 407, wherein said action probability distribution update comprises a inaction-penalty update.
421. The method of claim 407, wherein said action probability distribution is normalized.
422. The method of claim 407, wherein said selected processor action corresponds to the highest probability value within said action probability distribution.
423. The method of claim 407, wherein said selected processor action corresponds to a pseudo-random selection of a probability value within said action probability distribution.
424. The method of claim 407, wherein said processing device is a computer game, said identified user action is a player move, and said processor actions are game moves.
425. The method of claim 407, wherein said processing device is an educational toy, said identified user action is a child action, and said processor actions are toy actions.
426. The method of claim 407, wherein said processing device is a telephone system, said identified user action is a called phone number, and said processor actions are listed phone numbers.
427. The method of claim 407, wherein said processing device is a television channel control system, said identified user action is a watched television channel, and said processor actions are listed television channels.
428. The method of claim 407, wherein said action probability distribution is updated using a learning automaton.
429. A processing device having a function independent of determining an optimum action, comprising:
- an action selection module configured for selecting one of a plurality of processor actions, said action selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions, wherein said selected processor action affects said processing device function;
- an outcome evaluation module configured for determining an outcome of one or both of said identified user action and said selected processor action; and
- a probability update module configured for updating said action probability distribution based on said outcome.
430. The processing device of claim 429, wherein said outcome is of said identified user action.
431. The processing device of claim 429, wherein said outcome is of said selected processor action.
432. The processing device of claim 429, wherein said outcome is of said selected processor action relative to said identified user action.
433. The processing device of claim 429, wherein said processing device is a computer game.
434. The processing device of claim 429, wherein said processing device is an educational toy.
435. The processing device of claim 429, wherein said processing device is a telephone system.
436. The processing device of claim 429, wherein said processing device is a television channel control system.
437. The processing device of claim 429, wherein said probability update module is configured for using a learning automaton to update said action probability distribution.
438. A method of providing learning capability to a processing device having one or more objectives, comprising:
- identifying actions from a plurality of users;
- selecting one or more of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- determining one or more outcomes of one or both of said identified plurality of user actions and said selected one or more processor actions;
- updating said action probability distribution using one or more learning automatons based on said one or more outcomes; and
- modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
439. The method of claim 438, wherein said one or more outcome are of said identified plurality of user actions.
440. The method of claim 438, wherein said one or more outcome are of said selected one or more processor actions.
441. The method of claim 438, wherein said one or more outcome are of said selected one or more processor actions relative to said plurality of identified plurality of user actions.
442. The method of claim 438, wherein said selected one or more processor actions comprises a single processor action corresponding to said identified plurality of user actions.
443. The method of claim 438, wherein said selected one or more processor actions comprises a plurality of processor actions respectively corresponding to said identified plurality of user actions.
444. The method of claim 438, wherein said one or more outcomes comprises a single outcome corresponding to said identified plurality of user actions.
445. The method of claim 438, wherein said one or more outcomes comprises a plurality of outcomes respectively corresponding to said identified plurality of user actions.
446. The method of claim 438, wherein said action probability distribution is updated when a predetermined period of time has expired.
447. The method of claim 438, wherein said action probability distribution is updated in response to said identification of each user action.
448. The method of claim 438, wherein said selected processor action is selected in response to said identified plurality of user actions.
449. The method of claim 438, further comprising determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
450. The method of claim 449, wherein said one or more performance indexes comprises a single performance index corresponding to said identified plurality of user actions.
451. The method of claim 449, wherein said one or more performance indexes comprises a plurality of performance indexes respectively corresponding to said identified plurality of user actions.
452. The method of claim 438, wherein said modification comprises modifying said action selection.
453. The method of claim 438, wherein said modification comprises modifying said outcome determination.
454. The method of claim 438, wherein said modification comprises modifying said action probability distribution update.
455. The method of claim 438, wherein said modification comprises selecting one of a predetermined plurality of algorithms employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
456. The method of claim 438, wherein said modification comprises modifying a parameter of an algorithm employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
457. The method of claim 438, wherein outcome determination is performed only after several iterations of said identified user action identification and processor action selection.
458. The method of claim 438, wherein said probability distribution update is performed only after several iterations of said identified user action identification and processor action selection.
459. The method of claim 438, wherein said probability distribution update is performed only after several iterations of said identified user action identification, processor action selection, and outcome determination.
460. The method of claim 438, wherein said processing device is a computer game, said identified user actions are player moves, and said processor actions are game moves.
461. A method of providing learning capability to a processing device having one or more objectives, comprising:
- identifying actions from users divided amongst a plurality of user sets;
- for each of said user sets:
- selecting one or more of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- determining one or more outcomes of one or more actions from said each user set and said selected one or more processor actions;
- updating said action probability distribution using a learning automaton based on said one or more outcomes; and
- modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
462. The method of claim 461, wherein each user set comprises a single user.
463. The method of claim 461, wherein each user set comprises a plurality of users.
464. The method of claim 463, wherein said selected one or more processor actions comprises a single processor action corresponding to actions from said plurality of users.
465. The method of claim 463, wherein said selected one or more processor actions comprises a plurality of processor actions respectively corresponding to actions from said plurality of users.
466. The method of claim 463, wherein said one or more outcomes comprises a single outcome corresponding to actions from said plurality of users.
467. The method of claim 463, wherein said one or more outcomes comprises a plurality of outcomes respectively corresponding to actions from said plurality of users.
468. The method of claim 461, wherein said action probability distribution is updated when a predetermined period of time has expired.
469. The method of claim 461, wherein said action probability distribution is updated in response to said identification of each user action.
470. The method of claim 461, wherein said selected one or more processor actions is selected in response to said identified user actions.
471. The method of claim 461, further comprising determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
472. The method of claim 463, further comprising determining a single performance index indicative of a performance of said processing device relative to said one or more objectives, wherein said single performance index corresponds to said identified plurality of user actions and said modification is based on said single performance index.
473. The method of claim 463, further comprising determining a plurality of performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said plurality of performance indexes corresponds to said identified plurality of user actions and said modification is based on said plurality of performance indexes.
474. The method of claim 461, wherein said modification comprises modifying said action selection.
475. The method of claim 461, wherein said modification comprises modifying said outcome determination.
476. The method of claim 461, wherein said modification comprises modifying said action probability distribution update.
477. The method of claim 461, wherein said modification comprises selecting one of a predetermined plurality of algorithms employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
478. The method of claim 461, wherein said modification comprises modifying a parameter of an algorithm employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
479. The method of claim 461, wherein outcome determination is performed only after several iterations of said identified user action identification and processor action selection.
480. The method of claim 461, wherein said probability distribution update is performed only after several iterations of said identified user action identification and processor action selection.
481. The method of claim 461, wherein said probability distribution update is performed only after several iterations of said identified user action identification, processor action selection, and outcome determination.
482. The method of claim 461, wherein said processing device is a computer game, said identified user actions are player moves, and said processor actions are game moves.
483. The method of claim 461, wherein said processing device is a telephone system, said identified user actions are called phone numbers, and said processor actions are listed phone numbers.
484. A processing device having one or more objectives, comprising:
- a probabilistic learning module having a learning automaton configured for learning a plurality of processor actions in response to a plurality of actions performed by a plurality of users; and
- an intuition module configured for modifying a functionality of said probabilistic learning module based on said one or more objectives.
485. The processing device of claim 484, wherein said intuition module is further configured for determining one or more performance indexes indicative of a performance of said probabilistic learning module relative to said one or more objectives, and for modifying said probabilistic learning module functionality based on said one or more performance indexes.
486. The processing device of claim 485, wherein said one or more performance indexes comprises a single performance index corresponding to said plurality of users.
487. The processing device of claim 485, wherein said one or more performance indexes comprises a plurality of performance indexes respectively corresponding to said plurality of users.
488. The processing device of claim 484, wherein said intuition module is configured for selecting one of a predetermined plurality of algorithms employed by said learning module.
489. The processing device of claim 484, wherein said intuition module is configured for modifying a parameter of an algorithm employed by said learning module.
490. The processing device of claim 484, wherein said probabilistic learning module comprises:
- one or more action selection modules configured for selecting one or more of a plurality of processor actions, said action selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- one or more outcome evaluation modules configured for determining one or more outcomes of one or both of said identified plurality of user actions and said selected one or more processor actions; and
- a probability update module configured for updating said action probability distribution based on said one or more outcomes.
491. The processing device of claim 490, wherein said one or more outcome are of said identified plurality of user actions.
492. The processing device of claim 490, wherein said one or more outcome are of said selected one or more processor actions.
493. The processing device of claim 490, wherein said one or more outcome are of said selected one or more processor actions relative to said plurality of identified plurality of user actions.
494. The processing device of claim 490, wherein said selected one or more processor actions comprises a single processor action corresponding to said identified plurality of user actions.
495. The processing device of claim 490, wherein said selected one or more processor actions comprises a plurality of processor actions respectively corresponding to said identified plurality of user actions.
496. The processing device of claim 490, wherein said one or more outcomes comprises a single outcome corresponding to said identified plurality of user actions.
497. The processing device of claim 490, wherein said one or more outcomes comprises a plurality of outcomes respectively corresponding to said identified plurality of user actions.
498. The processing device of claim 490, wherein said intuition module is configured for modifying a functionality of said one or more action selection modules based on said one or more objectives.
499. The processing device of claim 490, wherein said intuition module is configured for modifying a functionality of said one or more outcome evaluation modules based on said one or more objectives.
500. The processing device of claim 490, wherein said intuition module is configured for modifying a functionality of said probability update module based on said one or more objectives.
501. The processing device of claim 490, further comprising:
- a server containing said one or more action selection modules, said one or more outcome evaluation modules, and said probability update module;
- a plurality of computers configuring for respectively generating said identified plurality of user actions; and
- a network configured for transmitting said identified plurality of user actions from said plurality of computers to said server and for transmitting said selected one or more processor actions from said server to said plurality of computers.
502. The processing device of claim 490, wherein said one or more action selection modules comprises a plurality of action selection modules, and said selected one or more processor actions comprises a plurality of processor actions, the processing device further comprising:
- a server containing said one or more outcome evaluation modules, and said probability update module;
- a plurality of computers configuring for respectively generating said identified plurality of user actions, said plurality of computers respectively containing said plurality of action selection modules; and
- a network configured for transmitting said identified plurality of user actions and said selected plurality of processor actions from said plurality of computers to said server.
503. The processing device of claim 490, wherein said one or more action selection modules comprises a plurality of action selection modules, said selected one or more processor actions comprises a plurality of processor actions, said one or more outcome evaluation modules comprises a plurality of outcome evaluation modules, and said one or more outcomes comprises a plurality of outcomes, the processing device further comprising:
- a server containing said probability update module;
- a plurality of computers configuring for respectively generating said identified plurality of user actions, said plurality of computers respectively containing said plurality of action selection modules and said plurality of outcome evaluation modules; and
- a network configured for transmitting said plurality of outcomes from said plurality of computers to said server.
504. The processing device of claim 484, wherein said plurality of users are divided amongst a plurality of user sets, and wherein said probabilistic learning module comprises:
- one or more action selection modules configured for, each user set, selecting one or more of a plurality of processor actions, said action selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- one or more outcome evaluation modules configured for, for said each user set, determining one or more outcomes of one or both of one or more user actions and said selected one or more processor actions; and
- one or more probability update modules configured for, for said each user set, updating said action probability distribution based on said one or more outcomes.
505. The processing device of claim 504, wherein said one or more outcome are of said identified plurality of user actions.
506. The processing device of claim 504, wherein said one or more outcome are of said selected one or more processor actions.
507. The processing device of claim 504, wherein said one or more outcome are of said selected one or more processor actions relative to said plurality of identified plurality of user actions.
508. The processing device of claim 504, wherein each user set comprises a single user.
509. The processing device of claim 504, wherein each user set comprises a plurality of users.
510. The processing device of claim 504, wherein said selected one or more processor actions comprises a single processor action corresponding to said identified plurality of user actions.
511. The processing device of claim 504, wherein said selected one or more processor actions comprises a plurality of processor actions respectively corresponding to said identified plurality of user actions.
512. The processing device of claim 504, wherein said intuition module is configured for modifying a functionality of said one or more action selection modules based on said one or more objectives.
513. The processing device of claim 504, wherein said intuition module is configured for modifying a functionality of said one or more outcome evaluation modules based on said one or more objectives.
514. The processing device of claim 504, wherein said intuition module is configured for modifying a functionality of said probability update module based on said one or more objectives.
515. The processing device of claim 504, further comprising:
- a server containing said one or more action selection modules, said one or more outcome evaluation modules, and said one or more probability update modules;
- a plurality of computers configuring for respectively generating said identified plurality of user actions; and
- a network configured for transmitting said identified plurality of user actions from said plurality of computers to said server and for transmitting said selected one or more processor actions from said server to said plurality of computers.
516. The processing device of claim 504, wherein said one or more action selection modules comprises a plurality of action selection modules, and said selected one or more processor actions comprises a plurality of processor actions, the processing device further comprising:
- a server containing said one or more outcome evaluation modules and said one or more probability update modules;
- a plurality of computers configuring for respectively generating said identified plurality of user actions, said plurality of computers respectively containing said plurality of action selection modules; and
- a network configured for transmitting said identified plurality of user actions and said selected plurality of processor actions from said plurality of computers to said server.
517. The processing device of claim 504, wherein said one or more action selection modules comprises a plurality of action selection modules, said selected one or more processor actions comprises a plurality of processor actions, said one or more outcome evaluation modules comprises a plurality of outcome evaluation modules, and said one or more outcomes comprises a plurality of outcomes, the processing device further comprising:
- a server containing said one or more probability update modules;
- a plurality of computers configuring for respectively generating said identified plurality of user actions, said plurality of computers respectively containing said plurality of action selection modules and said plurality of outcome evaluation modules; and
- a network configured for transmitting said plurality of outcomes from said plurality of computers to said server.
518. The processing device of claim 503, wherein said one or more action selection modules comprises a plurality of action selection modules, said selected one or more processor actions comprises a plurality of processor actions, said one or more outcome evaluation modules comprises a plurality of outcome evaluation modules, and said one or more outcomes comprises a plurality of outcomes, said one or more probability update modules comprises a plurality of update modules for updating said plurality of action probability distributions, the processing device further comprising:
- a server containing said a module for generating a centralized action probability distribution based on said plurality of action probability distributions, said centralized action probability distribution used to initialize a subsequent plurality of action probability distributions;
- a plurality of computers configuring for respectively generating said identified plurality of user actions, said plurality of computers respectively containing said plurality of action selection modules, said plurality of outcome evaluation modules, and said plurality of probability update modules; and
- a network configured for transmitting said plurality of action probability distributions from said plurality of computers to said server, and said centralized action probability distribution from said server to said plurality of computers.
519. A method of providing learning capability to a processing device having one or more objectives, comprising:
- identifying a plurality of user actions;
- selecting one or more of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- weighting said identified plurality of user actions;
- determining one or more outcomes of said plurality of weighted user actions; and
- updating said action probability distribution based on said one or more outcomes.
520. The method of claim 519, wherein said identified plurality of user actions is received from a plurality of users.
521. The method of claim 520, wherein said weighting is based on a skill level of said plurality of users.
522. The method of claim 519, wherein said one or more selected processor actions is selected in response to said identified plurality of user actions.
523. The method of claim 519, wherein said selected one or more processor actions comprises a single processor action corresponding to said identified plurality of user actions.
524. The method of claim 519, wherein said selected one or more processor actions comprises a plurality of processor actions respectively corresponding to said identified plurality of user actions.
525. The method of claim 519, wherein said one or more outcomes comprises a single outcome corresponding to said identified plurality of user actions.
526. The method of claim 519, wherein said one or more outcomes comprises a plurality of outcomes respectively corresponding to said identified plurality of user actions.
527. The method of claim 519, further comprising modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
528. The method of claim 527, further comprising determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
529. The method of claim 527, wherein said one or more performance indexes comprises a single performance index corresponding to said identified plurality of user actions.
530. The method of claim 527, wherein said one or more performance indexes comprises a plurality of performance indexes respectively corresponding to said identified plurality of user actions.
531. The method of claim 527, wherein said modification comprises modifying said action selection.
532. The method of claim 527, wherein said modification comprises modifying said outcome determination.
533. The method of claim 532, wherein said outcome determination modification comprises modifying said weighting of said identified plurality of user actions.
534. The method of claim 527, wherein said modification comprises modifying said action probability distribution update.
535. The method of claim 527, wherein said modification comprises selecting one of a predetermined plurality of algorithms employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
536. The method of claim 527, wherein said modification comprises modifying a parameter of an algorithm employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
537. The method of claim 519, wherein said action probability distribution is updated using a learning automaton.
538. The method of claim 519, wherein said processing device is a computer game, said identified user actions are player moves, and said processor actions are game moves.
539. A processing device having one or more objectives, comprising:
- an action selection module configured for selecting one or more of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- an outcome evaluation module configured for weighting a plurality of identified user actions, and for determining one or more outcomes of said plurality of weighted user actions; and
- a probability update module configured for updating said action probability distribution based on said outcome.
540. The processing device of claim 539, wherein said identified plurality of user actions is received from a plurality of users.
541. The processing device of claim 540, wherein said weighting is based on a skill level of said plurality of users.
542. The processing device of claim 539, wherein said action selection module is configured for selecting said one or more selected processor actions in response to said identified plurality of user actions.
543. The processing device of claim 539, wherein said selected one or more processor actions comprises a single processor action corresponding to said identified plurality of user actions.
544. The processing device of claim 539, wherein said selected one or more processor actions comprises a plurality of processor actions respectively corresponding to said identified plurality of user actions.
545. The processing device of claim 539, wherein said one or more outcomes comprises a single outcome corresponding to said identified plurality of user actions.
546. The processing device of claim 539, wherein said one or more outcomes comprises a plurality of outcomes respectively corresponding to said identified plurality of user actions.
547. The processing device of claim 539, further comprising an intuition module configured for modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
548. The processing device of claim 547, wherein said intuition module is further configured for determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
549. The processing device of claim 547, wherein said one or more performance indexes comprises a single performance index corresponding to said identified plurality of user actions.
550. The processing device of claim 547, wherein said one or more performance indexes comprises a plurality of performance indexes respectively corresponding to said identified plurality of user actions.
551. The processing device of claim 547, wherein said intuition module is configured for modifying a functionality of said action selection module based on said one or more objectives.
552. The processing device of claim 547, wherein said intuition module is configured for modifying a functionality of said outcome evaluation module based on said one or more objectives.
553. The processing device of claim 547, wherein said intuition module is configured for modifying a functionality of said probability update module based on said one or more objectives.
554. The processing device of claim 539, wherein said probability update module is configured for using a learning automaton to update said action probability distribution.
555. A method of providing learning capability to a processing device having one or more objectives, comprising:
- identifying a plurality of user actions;
- selecting one of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- determining a success ratio of said selected processor action relative to said identified plurality of user actions;
- comparing said determined success ratio to a reference success ratio;
- determining an outcome of said success ratio comparison; and
- updating said action probability distribution based on said outcome.
556. The method of claim 555, wherein said identified plurality of user actions is received from a plurality of users.
557. The method of claim 555, wherein said identified plurality of user actions is received from a single user.
558. The method of claim 555, wherein said reference success ratio is a simple majority.
559. The method of claim 555, wherein said reference success ratio is a minority.
560. The method of claim 555, wherein said reference success ratio is a super majority.
561. The method of claim 555, wherein said reference success ratio is a unanimity.
562. The method of claim 555, wherein said reference success ratio is an equality.
563. The method of claim 555, wherein said selected processor action is selected in response to said identified plurality of user actions.
564. The method of claim 555, further comprising modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
565. The method of claim 564, further comprising determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
566. The method of claim 564, wherein said modification comprises modifying said reference success ratio.
567. The method of claim 564, wherein said modification comprises modifying said action selection.
568. The method of claim 564, wherein said modification comprises modifying said outcome determination.
569. The method of claim 564, wherein said modification comprises modifying said action probability distribution update.
570. The method of claim 564, wherein said modification comprises selecting one of a predetermined plurality of algorithms employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
571. The method of claim 564, wherein said modification comprises modifying a parameter of an algorithm employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
572. The method of claim 555, wherein said action probability distribution is updated using a learning automaton.
573. The method of claim 555, wherein said processing device is a computer game, said identified user actions are player moves, and said processor actions are game moves.
574. A processing device having one or more objectives, comprising:
- an action selection module configured for selecting one of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- an outcome evaluation module configured for determining a success ratio of said selected processor action relative to a plurality of user actions, for comparing said determined success ratio to a reference success ratio, and for determining an outcome of said success ratio comparison; and
- a probability update module configured for updating said action probability distribution based on said outcome.
575. The processing device of claim 574, wherein said identified plurality of user actions is received from a plurality of users.
576. The processing device of claim 574, wherein said identified plurality of user actions is received from a single user.
577. The processing device of claim 574, wherein said reference success ratio is a simple majority.
578. The processing device of claim 574, wherein said reference success ratio is a minority.
579. The processing device of claim 574, wherein said reference success ratio is a super majority.
580. The processing device of claim 574, wherein said reference success ratio is a unanimity.
581. The processing device of claim 574, wherein said reference success ratio is an equality.
582. The processing device of claim 574, wherein said action selection module is configured for selecting said processor action in response to said identified plurality of user actions.
583. The processing device of claim 574, further comprising an intuition module configured for modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
584. The processing device of claim 583, wherein said intuition module is further configured for determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
585. The processing device of claim 583, wherein said one or more performance indexes comprises a single performance index corresponding to said identified plurality of user actions.
586. The processing device of claim 583, wherein said one or more performance indexes comprises a plurality of performance indexes respectively corresponding to said identified plurality of user actions.
587. The processing device of claim 583, wherein said intuition module is configured for modifying a functionality of said action selection module based on said one or more objectives.
588. The processing device of claim 583, wherein said intuition module is configured for modifying a functionality of said outcome evaluation module based on said one or more objectives.
589. The processing device of claim 583, wherein said intuition module is configured for modifying a functionality of said probability update module based on said one or more objectives.
590. The processing device of claim 574, wherein said probability update module is configured for using a learning automaton to update said action probability distribution.
591. A method of providing learning capability to a processing device having one or more objectives, comprising:
- identifying actions from a plurality of users;
- selecting one of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- determining if said selected processor action has a relative success level for a majority of said plurality of users;
- determining an outcome of said success determination; and
- updating said action probability distribution based on said outcome.
592. The method of claim 591, wherein said reference success level is a greatest success.
593. The method of claim 591, wherein said reference success level is a least success.
594. The method of claim 591, wherein said reference success level is an average success.
595. The method of claim 591, further comprising maintaining separate action probability distributions for said plurality of users, wherein said relative success level of said selected processor action is determined from said separate action probability distributions.
596. The method of claim 591, further comprising maintaining an estimator success table for said plurality of users, wherein said relative success level of said selected processor action is determined from said estimator table.
597. The method of claim 591, wherein said selected processor action is selected in response to said identified plurality of user actions.
598. The method of claim 591, further comprising modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
599. The method of claim 598, further comprising determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
600. The method of claim 598, wherein said modification comprises modifying said relative success level.
601. The method of claim 598, wherein said modification comprises modifying said action selection.
602. The method of claim 598, wherein said modification comprises modifying said outcome determination.
603. The method of claim 598, wherein said modification comprises modifying said action probability distribution update.
604. The method of claim 598, wherein said modification comprises selecting one of a predetermined plurality of algorithms employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
605. The method of claim 598, wherein said modification comprises modifying a parameter of an algorithm employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
606. The method of claim 591, wherein said action probability distribution is updated using a learning automaton.
607. The method of claim 591, wherein said processing device is a computer game, said identified user actions are player moves, and said processor actions are game moves.
608. A processing device having one or more objectives, comprising:
- an action selection module configured for selecting one of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions;
- an outcome evaluation module configured for determining if said selected processor action has a relative success level for a majority of a plurality of users, and for determining an outcome of said success determination; and
- a probability update module configured for updating said action probability distribution based on said outcome.
609. The processing device of claim 608, wherein said reference success level is a greatest success.
610. The processing device of claim 608, wherein said reference success level is a least success.
611. The processing device of claim 608, wherein said reference success level is an average success.
612. The processing device of claim 608, wherein said probability update module is further configured for maintaining separate action probability distributions for said plurality of users, and said outcome evaluation module is configured for determining said relative success level of said selected processor action from said separate action probability distributions.
613. The processing device of claim 608, wherein said outcome evaluation module is further configured for maintaining an estimator success table for said plurality of users, and for determining said relative success level of said selected processor action from said estimator table.
614. The processing device of claim 608, wherein said action selection module is configured for selecting said selected processor action in response to said identified plurality of user actions.
615. The processing device of claim 608, further comprising an intuition module configured for modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
616. The processing device of claim 615, wherein said intuition module is further configured for determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
617. The processing device of claim 615, wherein said one or more performance indexes comprises a single performance index corresponding to said identified plurality of user actions.
618. The processing device of claim 615, wherein said one or more performance indexes comprises a plurality of performance indexes respectively corresponding to said identified plurality of user actions.
619. The processing device of claim 615, wherein said intuition module is configured for modifying a functionality of said action selection module based on said one or more objectives.
620. The processing device of claim 615, wherein said intuition module is configured for modifying a functionality of said outcome evaluation module based on said one or more objectives.
621. The processing device of claim 615, wherein said intuition module is configured for modifying a functionality of said probability update module based on said one or more objectives.
622. The processing device of claim 608, wherein said probability update module is configured for using a learning automaton to update said action probability distribution.
623. A method of providing learning capability to a processing device having one or more objectives, comprising:
- selecting one or more of a plurality of processor actions that are respectively linked to a plurality of user parameters, said selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of linked processor actions;
- linking said one or more selected process actions with one or more of said plurality of user parameters;
- determining one or more outcomes of said one or more linked processor actions; and
- updating said action probability distribution based on said one or more outcomes.
624. The method of claim 623, wherein said plurality of user parameters comprises a plurality of user actions.
625. The method of claim 623, wherein said plurality of user parameters comprises a plurality of users.
626. The method of claim 623, wherein said plurality of processor actions is linked to another plurality of user parameters.
627. The method of claim 626, wherein said plurality of user parameters comprises a plurality of user actions, and said other plurality of user parameters comprises a plurality of users.
628. The method of claim 623, further comprising identifying one or more user actions.
629. The method of claim 628, wherein said selected one or more processor actions is selected in response to said one or more user actions.
630. The method of claim 628, wherein said one or more user actions comprises a plurality of user actions.
631. The method of claim 630, wherein said selected one or more processor actions comprises a single processor action corresponding to said identified plurality of user actions.
632. The method of claim 630, wherein said selected one or more processor actions comprises a plurality of processor actions respectively corresponding to said identified plurality of user actions.
633. The method of claim 630, wherein said one or more outcomes comprises a single outcome corresponding to said identified plurality of user actions.
634. The method of claim 630, wherein said one or more outcomes comprises a plurality of outcomes respectively corresponding to said identified plurality of user actions.
635. The method of claim 623, further comprising modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
636. The method of claim 635, further comprising determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
637. The method of claim 635, wherein said modification comprises modifying said action selection.
638. The method of claim 635, wherein said modification comprises modifying said outcome determination.
639. The method of claim 635, wherein said modification comprises modifying said action probability distribution update.
640. The method of claim 635, wherein said modification comprises selecting one of a predetermined plurality of algorithms employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
641. The method of claim 635, wherein said modification comprises modifying a parameter of an algorithm employed by said one or more of said processor action selection, said outcome determination, and said action probability distribution update.
642. The method of claim 623, wherein said action probability distribution is updated using a learning automaton.
643. The method of claim 623, wherein said processing device is a computer game, said one or more user actions are one or more player moves, and said processor actions are game moves.
644. A processing device having one or more objectives, comprising:
- an action selection module configured for selecting one or more of a plurality of processor actions that are respectively linked to a plurality of user parameters, said selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of linked processor actions;
- an outcome evaluation module configured for linking said one or more selected process actions with one or more of said plurality of user parameters, and for determining one or more outcomes of said one or more linked processor actions and one or more user actions; and
- a probability update module configured for updating said action probability distribution based on said one or more outcomes.
645. The processing device of claim 644, wherein said plurality of user parameters comprises a plurality of user actions.
646. The processing device of claim 644, wherein said plurality of user parameters comprises a plurality of users.
647. The processing device of claim 644, wherein said outcome evaluation module is configured for linking said plurality of processor actions to another plurality of user parameters.
648. The processing device of claim 647, wherein said plurality of user parameters comprises a plurality of user actions, and said other plurality of user parameters comprises a plurality of users.
649. The processing device of claim 644, wherein said action selection module is configured for selecting said selected one or more processor actions in response to said one or more user actions.
650. The processing device of claim 644, wherein said one or more user actions comprises a plurality of user actions.
651. The processing device of claim 650, wherein said selected one or more processor actions comprises a single processor action corresponding to said identified plurality of user actions.
652. The processing device of claim 650, wherein said selected one or more processor actions comprises a plurality of processor actions respectively corresponding to said identified plurality of user actions.
653. The processing device of claim 650, wherein said one or more outcomes comprises a single outcome corresponding to said identified plurality of user actions.
654. The processing device of claim 650, wherein said one or more outcomes comprises a plurality of outcomes respectively corresponding to said identified plurality of user actions.
655. The processing device of claim 644, further comprising an intuition module configured for modifying one or more of said processor action selection, said outcome determination, and said action probability distribution update based on said one or more objectives.
656. The processing device of claim 655, wherein said intuition module is further configured for determining one or more performance indexes indicative of a performance of said processing device relative to said one or more objectives, wherein said modification is based on said one or more performance indexes.
657. The processing device of claim 655, wherein said one or more performance indexes comprises a single performance index corresponding to said identified plurality of user actions.
658. The processing device of claim 655, wherein said one or more performance indexes comprises a plurality of performance indexes respectively corresponding to said identified plurality of user actions.
659. The processing device of claim 655, wherein said intuition module is configured for modifying a functionality of said action selection module based on said one or more objectives.
660. The processing device of claim 655, wherein said intuition module is configured for modifying a functionality of said outcome evaluation module based on said one or more objectives.
661. The processing device of claim 655, wherein said intuition module is configured for modifying a functionality of said probability update module based on said one or more objectives.
662. The processing device of claim 644, wherein said probability update module is configured for using a learning automaton to update said action probability distribution.
663. A method of providing learning capability to a processing device having an objective, comprising:
- generating a list containing a plurality of listed items with an associated item probability distribution comprising a plurality of probability values corresponding to said plurality of listed items;
- selecting one or more items from said plurality of listed items based on said item probability distribution;
- determining a performance index indicative of a performance of said processing device relative to said objective; and
- modifying said item probability distribution based on said performance index.
664. The method of claim 663, further comprising identifying an action associated item; and
- determining if said identified item matches any listed item contained in said list, wherein said performance index is derived from said matching determination.
665. The method of claim 663, wherein said selected one or more items comprises a plurality of selected items.
666. The method of claim 663, wherein said selected one or more items comprises a single selected item.
667. The method of claim 663, wherein said selected one or more items corresponds to the highest one or more probability values in said item probability distribution.
668. The method of claim 663, further comprising placing said plurality of listed items in an order according to corresponding probability values.
669. The method of claim 663, wherein said item probability distribution is modified by updating said item probability distribution.
670. The method of claim 669, wherein said item probability distribution update comprises a reward-inaction update.
671. The method of claim 663, wherein said item probability distribution is modified by increasing a probability value.
672. The method of claim 663, wherein said item probability distribution is modified by adding a probability value.
673. The method of claim 672, wherein said item probability distribution is modified by replacing a probability value with said added probability value.
674. The method of claim 663, wherein said performance index is instantaneous.
675. The method of claim 663, wherein said performance index is cumulative.
676. The method of claim 663, wherein said item probability distribution is normalized.
677. The method of claim 664, wherein said item probability distribution is modified by updating it if said identified item matches said any listed item.
678. The method of claim 677, wherein said item probability distribution update comprises a reward-inaction update.
679. The method of claim 678, wherein a corresponding probability value is rewarded if said identified item matches said any listed item.
680. The method of claim 677, further comprising adding a listed item corresponding to said identified item to said list if said identified item does not match said any listed item, wherein said item probability distribution is modified by adding a probability value corresponding to said added listed item to said item probability distribution.
681. The method of claim 680, wherein another item on said list is replaced with said added listed item, and another probability value corresponding to said replaced listed item is replaced with said added probability value.
682. The method of claim 663, wherein said item probability distribution is modified by updating it only if said identified item matches an item within said selected one or more items.
683. The method of claim 682, wherein said item probability distribution update comprises a reward-inaction update.
684. The method of claim 683, wherein a corresponding probability value is rewarded if said identified item matches said any listed item.
685. The method of claim 682, wherein said item probability distribution is modified by increasing a corresponding probability value if said identified item matches a listed item that does not correspond to an item in said selected one or more items.
686. The method of claim 682, further comprising adding a listed item corresponding to said identified item to said list if said identified item does not match said any listed item, wherein said item probability distribution is modified by adding a probability value corresponding to said added listed item to said item probability distribution.
687. The method of claim 686, wherein another item on said list is replaced with said added listed item, and another probability value corresponding to said replaced listed item is replaced with said added probability value.
688. The method of claim 663, wherein said processing device comprises a phone, said identified item comprises a phone number associated with a phone call, said listed items comprises listed phone numbers, and said item probability distribution comprises a phone number probability distribution.
689. The method of claim 663, wherein said processing device is a television, said identified item comprises a television channel that is watched, said listed items comprises listed television channels, and said item probability distribution comprises a television channel probability distribution.
690. The method of claim 663, further comprising:
- generating another list containing another plurality of listed items with another associated item probability distribution comprising another plurality of probability values corresponding to said other plurality of listed items; and
- selecting another one or more items from said other plurality of items based on said other item probability distribution.
691. The method of claim 690, further comprising:
- identifying another action associated item; and
- determining if said identified item matches any listed item contained in said list;
- identifying another action associated item; and
- determining if said other identified item matches any listed item contained in said other list;
- wherein said performance index is derived from said matching determinations.
692. The method of claim 690, further comprising:
- identifying an action associate item;
- determining current temporal information;
- selecting one of said list and said other list based on said current temporal information determination; and
- determining if said identified items matches any listed item contained in said selected list, wherein said performance index is derived from said determination.
693. The method of claim 692, wherein said temporal information is a day of the week.
694. The method of claim 692, wherein said temporal information is a time of the day.
695. The method of claim 663, wherein said item probability distribution is updated using a learning automaton.
696. The method of claim 663, wherein said item probability distribution is purely frequency based.
697. The method of claim 696, wherein said item probability distribution is based on a moving average.
698. The method of claim 663, wherein said list is one of a plurality of like lists corresponding to a plurality of users, the method further comprising determining which user performed said identified item, wherein said list corresponds with said determined user.
699. The method of claim 698, wherein said user determination is based on the operation of one of a plurality of keys associated with said processing device.
700. A processing device having an objective, comprising:
- a probabilistic learning module configured for learning a plurality of favorite items of a user in response to identified user items; and
- an intuition module configured for modifying a functionality of said probabilistic learning module based on said objective.
701. The processing device of claim 700, wherein said probabilistic learning module is further configured for determining a performance index indicative of a performance of said probabilistic learning module relative to said objective, and said intuition module is configured for modifying said probabilistic learning module functionality based on said performance index.
702. The processing device of claim 700, wherein said probabilistic learning module comprises:
- an item selection module configured for selecting said one or more of said plurality of favorite items from a plurality of items contained in a list, said selection being based on an item probability distribution comprising a plurality of probability values corresponding to said plurality of listed items;
- an outcome evaluation module configured for determining if an identified item matches any listed item contained in said list; and
- a probability update module, wherein said intuition module is configured for modifying said probability update module based on said matching determination.
703. The processing device of claim 702, wherein said one or more favorite items comprises a plurality of favorite items.
704. The processing device of claim 702, wherein said one or more favorite items comprises a single favorite item.
705. The processing device of claim 702, wherein said one or more favorite items correspond to one or more of the highest probability values in said item probability distribution.
706. The processing device of claim 702, wherein said item selection module is further configured for placing said plurality of listed items in an order according to corresponding probability values.
707. The processing device of claim 702, wherein said intuition module is configured for modifying said probability update module by directing it to update said item probability distribution if said identified item matches said any listed item.
708. The processing device of claim 707, wherein said probability update module is configured for updating said item probability distribution using a reward-inaction algorithm.
709. The processing device of claim 708, wherein said probability update module is configured for rewarding a corresponding probability value.
710. The processing device of claim 707, wherein said intuition module is configured for modifying said probability update module by adding a listed item corresponding to said identified item to said list and adding a probability value corresponding to said added listed items to said item probability distribution if said identified item does not match said any listed item.
711. The processing device of claim 710, wherein another item on said list is replaced with said added listed item, and another probability value corresponding to said replaced listed item is replaced with said added probability value.
712. The processing device of claim 702, wherein said intuition module is configured for modifying said probability update module by directing it to update said item probability distribution only if said identified item matches a listed item corresponding to one of said favorite items.
713. The processing device of claim 712, wherein said probability update module is configured for updating said item probability distribution using a reward-inaction algorithm.
714. The processing device of claim 713, wherein said probability update module is configured for rewarding a corresponding probability value.
715. The processing device of claim 712, wherein said intuition module is configured for modifying said probability update module by increasing a corresponding probability value if said identified item matches a listed item that does not correspond to one of said one or more favorite items.
716. The processing device of claim 712, wherein said intuition module is configured for modifying said probability update module by adding a listed item corresponding to said identified item to said list and adding a probability value corresponding to said added listed item to said item probability distribution if said identified item does not match said any listed item.
717. The processing device of claim 716, wherein another item on said list is replaced with said added listed item, and another probability value corresponding to said replaced listed item is replaced with said added probability value.
718. The processing device of claim 700, wherein said performance index is instantaneous.
719. The processing device of claim 700, wherein said performance index is cumulative.
720. The processing device of claim 700, wherein said probabilistic learning module comprises a learning automaton.
721. The processing device of claim 700, wherein said probabilistic learning module is purely frequency-based.
722. The processing device of claim 700, wherein said learning module and said intuition module are self-contained in a single device.
723. The processing device of claim 700, wherein said learning module and said intuition module are contained in a telephone, and said items are phone numbers.
724. The processing device of claim 700, wherein said learning module and said intuition module are contained in television channel control system, and said items are television channels.
725. The processing device of claim 702, further comprising a favorite item function key the operation of which prompts said item selection module to select said one or more items.
726. The processing device of claim 702, wherein said list is one of a plurality of like lists corresponding to a plurality of users, and said item selection module is further configured for determining which user performed said identified item, wherein said list corresponds with said determined user.
727. The processing device of claim 726, further comprising a plurality of user function keys, wherein said user determination is based on the operation of one of said plurality of user function keys.
728. A method of providing learning capability to a processing device having an objective, comprising:
- generating a plurality of lists respectively corresponding to a plurality of item parameter values, each of said plurality of lists containing a plurality of listed items with an associated item probability distribution comprising a plurality of probability values corresponding to said plurality of listed items;
- selecting a list corresponding to a parameter value exhibited by a currently identified action associated item; and
- in said selected list, selecting one or more listed items from said plurality of listed items based on said item probability distribution;
- determining a performance index indicative of a performance of said processing device relative to said objective; and
- modifying said item probability distribution based on said performance index.
729. The method of claim 728, further comprising:
- identifying an action associated item exhibiting a parameter value;
- selecting a list corresponding to said identified parameter value; and
- determining if said identified item matches any listed item contained in said selected list, wherein said performance index is based on said matching determination.
730. The method of claim 728, wherein said selected one or more listed items comprises a plurality of selected items.
731. The method of claim 728, wherein said selected one or more listed items comprises a single selected item.
732. The method of claim 728, wherein said selected one or more listed items corresponds to the highest one or more probability values in said item probability distribution.
733. The method of claim 728, further comprising placing said plurality of listed items in an order according to corresponding probability values.
734. The method of claim 728, wherein said item probability distribution is modified by updating said item probability distribution.
735. The method of claim 734, wherein said item probability distribution update comprises a reward-inaction update.
736. The method of claim 728, wherein said item probability distribution is modified by increasing a probability value.
737. The method of claim 728, wherein said item probability distribution is modified by adding a probability value.
738. The method of claim 737, wherein said item probability distribution is modified by replacing a probability value with said added probability value.
739. The method of claim 728, wherein said performance index is instantaneous.
740. The method of claim 728, wherein said performance index is cumulative.
741. The method of claim 728, wherein said item probability distribution is modified by updating it if said identified item matches said any listed item.
742. The method of claim 741, wherein said item probability distribution update comprises a reward-inaction update.
743. The method of claim 742, wherein a corresponding probability value is rewarded if said identified item matches said any listed item.
744. The method of claim 741, further comprising adding a listed item corresponding to said identified item to said selected list if said identified item does not match said any listed item, wherein said item probability distribution is modified by adding a probability value corresponding to said added listed item to said item probability distribution.
745. The method of claim 744, wherein another listed item on said selected list is replaced with said added listed item, and another probability value corresponding to said replaced listed item is replaced with said added probability value.
746. The method of claim 728, wherein said item probability distribution is modified by updating it only if said identified item matches a listed item within said selected one or more items.
747. The method of claim 746, wherein said item probability distribution update comprises a reward-inaction update.
748. The method of claim 747, wherein a corresponding probability value is rewarded if said identified item matches said any listed item.
749. The method of claim 746, wherein said item probability distribution is modified by increasing a corresponding probability value if said identified item matches a listed item that does not correspond to an item in said selected one or more items.
750. The method of claim 746, further comprising adding a listed item corresponding to said identified item to said selected list if said identified item does not match said any listed item, wherein said item probability distribution is modified by adding a probability value corresponding to said added listed item to said item probability distribution.
751. The method of claim 750, wherein another listed item on said selected list is replaced with said added listed item, and another probability value corresponding to said replaced listed item is replaced with said added probability value.
752. The method of claim 728, wherein said processing device comprises a phone, and said item is a phone number.
753. The method of claim 728, wherein said processing device is a television channel control system and said item is a television channel.
754. The method of claim 728, wherein said item probability distribution is updated using a learning automaton.
755. The method of claim 728, wherein said item probability distribution is purely frequency based.
756. The method of claim 755, wherein said item probability distribution is based on a moving average.
757. A method of providing learning capability to a phone number calling system having an objective of anticipating called phone numbers, comprising:
- generating a phone list containing at least a plurality of listed phone numbers and a phone number probability distribution comprising a plurality of probability values corresponding to said plurality of listed phone numbers;
- selecting a set of phone numbers from said plurality of listed phone numbers based on said phone number probability distribution;
- determining a performance index indicative of a performance of said phone number calling system relative to said objective; and
- modifying said phone number probability distribution based on said performance index.
758. The method of claim 757, further comprising:
- identifying a phone number associated with a phone call; and
- determining if said identified phone number matches any listed phone number contained in said phone number list, wherein said performance index is derived from said matching determination.
759. The method of claim 757, wherein said selected phone number set is communicated to a user of said phone number calling system.
760. The method of claim 759, wherein said selected phone number set is displayed to said user.
761. The method of claim 757, wherein said selected phone number set comprises a plurality of selected phone numbers.
762. The method of claim 757, further comprising selecting a phone number from said selected phone number set to make a phone call.
763. The method of claim 757, wherein said selected phone number set corresponds to the highest probability values in said phone number probability distribution.
764. The method of claim 757, further comprising placing said selected phone number set in an order according to corresponding probability values.
765. The method of claim 757, wherein said identified phone number is associated with an outgoing phone call.
766. The method of claim 757, wherein said identified phone number is associated with an incoming phone call.
767. The method of claim 757, wherein said phone number probability distribution is modified by updating said phone number probability distribution.
768. The method of claim 767, wherein said phone number probability distribution update comprises a reward-inaction update.
769. The method of claim 757, wherein said phone number probability distribution is modified by increasing a probability value.
770. The method of claim 757, wherein said phone number probability distribution is modified by adding a probability value.
771. The method of claim 770, wherein said phone number probability distribution is modified by replacing a probability value with said added probability value.
772. The method of claim 757, wherein said plurality of probability values correspond to all phone numbers within said phone number list.
773. The method of claim 757, wherein said plurality of probability values correspond only to said plurality of phone numbers.
774. The method of claim 757, wherein said performance index is instantaneous.
775. The method of claim 757, wherein said performance index is cumulative.
776. The method of claim 757, wherein said phone number probability distribution is normalized.
777. The method of claim 758, wherein said phone number probability distribution is modified by updating it if said identified phone number matches said any listed phone number.
778. The method of claim 777, wherein said phone number probability distribution update comprises a reward-inaction update.
779. The method of claim 778, wherein a corresponding probability value is rewarded if said identified phone number matches said any listed phone number.
780. The method of claim 777, further comprising adding a listed phone number corresponding to said identified phone number to said phone list if said identified phone number does not match said any listed phone number, wherein said phone number probability distribution is modified by adding a probability value corresponding to said added listed phone number to said phone number probability distribution.
781. The method of claim 780, wherein another phone number on said phone list is replaced with said added listed phone number, and another probability value corresponding to said replaced listed phone number is replaced with said added probability value.
782. The method of claim 758, wherein said phone number probability distribution is modified by updating it only if said identified phone number matches a phone number within said selected phone number set.
783. The method of claim 782, wherein said phone number probability distribution update comprises a reward-inaction update.
784. The method of claim 783, wherein a corresponding probability value is rewarded if said identified phone number matches said any listed phone number.
785. The method of claim 782, wherein said phone number probability distribution is modified by increasing a corresponding probability value if said identified phone number matches a listed phone number that does not correspond to a phone number in said selected phone number set.
786. The method of claim 782, further comprising adding a listed phone number corresponding to said identified phone number to said phone list if said identified phone number does not match said any listed phone number, wherein said phone number probability distribution is modified by adding a probability value corresponding to said added listed phone number to said phone number probability distribution.
787. The method of claim 786, wherein another phone number on said phone list is replaced with said added listed phone number, and another probability value corresponding to said replaced listed phone number is replaced with said added probability value.
788. The method of claim 757, wherein said phone number calling system comprises a phone.
789. The method of claim 757, wherein said phone number calling system comprises a mobile phone.
790. The method of claim 757, further comprising:
- generating another phone list containing at least another plurality of listed phone numbers and a phone number probability distribution comprising a plurality of probability values corresponding to said other plurality of listed phone numbers; and
- selecting another set of phone numbers from said other plurality of phone numbers based on said other phone number probability distribution.
791. The method of claim 790, further comprising:
- identifying a phone number associated with a phone call; and
- determining if said identified phone number matches any listed phone number contained in said phone number list;
- identifying another phone number associated with another phone call; and
- determining if said other identified phone number matches any listed phone number contained in said other phone number list;
- wherein said performance index is derived from said matching determinations.
792. The method of claim 790, further comprising:
- identifying a phone number associated with a phone call;
- determining the current day of the week;
- selecting one of said phone list and said other phone list based on said current day determination; and
- determining if said identified phone number matches any listed phone number contained in said selected phone number list, wherein said performance index is derived from said determination.
793. The method of claim 790, further comprising:
- identifying a phone number associated with a phone call;
- determining a current time of the day;
- selecting one of said phone list and said other phone list based on said current time determination; and
- determining if said identified phone number matches any listed phone number contained in said selected phone number list, wherein said performance index is derived from said matching determination.
794. The method of claim 757, wherein said phone number probability distribution is updated using a learning automaton.
795. The method of claim 757, wherein said phone number probability distribution is purely frequency based.
796. The method of claim 795, wherein said phone number probability distribution is based on a moving average.
797. A phone number calling system having an objective of anticipating called phone numbers, comprising:
- a probabilistic learning module configured for learning favorite phone numbers of a user in response to phone calls; and
- an intuition module configured for modifying a functionality of said probabilistic learning module based on said objective.
798. The phone number calling system of claim 797, wherein said probabilistic learning module is further configured for determining a performance index indicative of a performance of said probabilistic learning module relative to said objective, and said intuition module is configured for modifying said probabilistic learning module functionality based on said performance index.
799. The phone number calling system of claim 797, further comprising a display for displaying said favorite phone numbers.
800. The phone number calling system of claim 797, further comprising one or more selection buttons configured for selecting one of said favorite phone numbers to make a phone call.
801. The phone number calling system of claim 797, wherein said phone calls are outgoing phone calls.
802. The phone number calling system of claim 797, wherein said phone calls are incoming phone calls.
803. The phone number calling system of claim 797, wherein said probabilistic learning module comprises:
- a phone number selection module configured for selecting said favorite phone numbers from a plurality of phone numbers based on a phone number probability distribution comprising a plurality of probability values corresponding to said plurality of listed phone numbers;
- an outcome evaluation module configured for determining if an identified phone number associated with a phone call matches any listed phone number contained in said phone number list; and
- a probability update module, wherein said intuition module is configured for modifying said probability update module based on said matching determinations.
804. The phone number calling system of claim 803, wherein said favorite phone numbers correspond to the highest probability values in said phone number probability distribution.
805. The phone number calling system of claim 803, wherein said phone number selection module is further configured for placing said favorite numbers in an order according to corresponding probability values.
806. The phone number calling system of claim 803, wherein said intuition module is configured for modifying said probability update module by directing it to update said phone number probability distribution if said identified phone number matches said any listed phone number.
807. The phone number calling system of claim 806, wherein said probability update module is configured for updating said phone number probability distribution using a reward-inaction algorithm.
808. The phone number calling system of claim 807, wherein said probability update module is configured for rewarding a corresponding probability value.
809. The phone number calling system of claim 806, wherein said intuition module is configured for modifying said probability update module by adding a listed phone number corresponding to said identified phone number to said phone list and adding a probability value corresponding to said added listed phone number to said phone number probability distribution if said identified phone number does not match said any listed phone number.
810. The phone number calling system of claim 809, wherein another phone number on said phone list is replaced with said added listed phone number, and another probability value corresponding to said replaced listed phone number is replaced with said added probability value.
811. The phone number calling system of claim 803, wherein said intuition module is configured for modifying said probability update module by directing it to update said phone number probability distribution only if said identified phone number matches a listed phone number corresponding to one of said favorite phone numbers.
812. The phone number calling system of claim 811, wherein said probability update module is configured for updating said phone number probability distribution using a reward-inaction algorithm.
813. The phone number calling system of claim 812, wherein said probability update module is configured for rewarding a corresponding probability value.
814. The phone number calling system of claim 811, wherein said intuition module is configured for modifying said probability update module by increasing a corresponding probability value if said identified phone number matches a listed phone number that does not correspond to one of said favorite phone numbers.
815. The phone number calling system of claim 811, wherein said intuition module is configured for modifying said probability update module by adding a listed phone number corresponding to said identified phone number to said phone list and adding a probability value corresponding to said added listed phone number to said phone number probability distribution if said identified phone number does not match said any listed phone number.
816. The phone number calling system of claim 815, wherein another phone number on said phone list is replaced with said added listed phone number, and another probability value corresponding to said replaced listed phone number is replaced with said added probability value.
817. The phone number calling system of claim 803, wherein said plurality of probability values correspond to all phone numbers within said phone number list.
818. The phone number calling system of claim 803, wherein said plurality of probability values correspond only to said plurality of listed phone numbers.
819. The phone number calling system of claim 798, wherein said performance index is instantaneous.
820. The phone number calling system of claim 798, wherein said performance index is cumulative.
821. The phone number calling system of claim 797, wherein said favorite phone numbers are divided into first and second favorite phone number lists, and said probabilistic learning module is configured for learning said first favorite phone number list in response to phone calls during a first time period, and for learning said second favorite phone number list in response to phone calls during a second time period.
822. The phone number calling system of claim 821, wherein said first time period includes weekdays, and said second time period includes weekends.
823. The phone number calling system of claim 821, wherein said first time period includes days, and said second time period includes evenings.
824. The phone number calling system of claim 797, wherein said probabilistic learning module comprises a learning automaton.
825. The phone number calling system of claim 797, wherein said probabilistic learning module is purely frequency-based.
826. The phone number calling system of claim 797, wherein said learning module and said intuition module are self-contained in a single device.
827. The phone number calling system of claim 797, wherein said learning module and said intuition module are contained in a telephone.
828. The phone number calling system of claim 827, wherein said telephone is a mobile telephone.
829. The phone number calling system of claim 797, wherein said learning module and said intuition module are contained in a server.
830. The phone number calling system of claim 797, wherein said learning module and said intuition module are distributed within a server and a phone.
831. A method of providing learning capability to a phone number calling system, comprising:
- identifying a plurality of phone numbers associated with a plurality of phone calls;
- maintaining a phone list containing said plurality of phone numbers and a plurality of priority values respectively associated with said plurality of phone numbers;
- selecting a set of phone numbers from said plurality of listed phone numbers based on said plurality of priority values;
- communicating said phone number set to a user.
832. The method of claim 831, further comprising updating a phone number probability distribution containing said plurality of priority values using a learning automaton.
833. The method of claim 831, further comprising updating a phone number probability distribution containing said plurality of priority values based purely on the frequency of said plurality of phone numbers.
834. The method of claim 833, wherein each of said plurality of priority values is based on a total number of times said associated phone number is identified during a specified time period.
835. The method of claim 831, wherein said selected phone number set is displayed to said user.
836. The method of claim 831, wherein said selected phone number set comprises a plurality of selected phone numbers.
837. The method of claim 831, further comprising selecting a phone number from said selected phone number set to make a phone call.
838. The method of claim 831, wherein said selected phone number set corresponds to the highest priority values.
839. The method of claim 831, further comprising placing said selected phone number set in an order according to corresponding priority values.
840. The method of claim 831, wherein said plurality of phone numbers is associated with outgoing phone calls.
841. The method of claim 831, wherein said plurality of phone numbers is associated with incoming phone calls.
842. The method of claim 831, wherein said phone number calling system comprises a phone.
843. The method of claim 831, wherein said phone number calling system comprises a mobile phone.
844. A method of providing learning capability to a television channel control system having an objective of anticipating watched television channels, comprising:
- generating a list containing a plurality of listed television channels with an associated television channel probability distribution comprising a plurality of probability values corresponding to said plurality of listed television channels;
- selecting one or more television channels from said plurality of listed television channels based on said television channel probability distribution;
- determining a performance index indicative of a performance of said processing device relative to said objective; and
- modifying said television channel probability distribution based on said performance index.
845. The method of claim 844, wherein said one or more television channels is selected in response to an operation of a function key.
846. The method of claim 844, further comprising:
- identifying a watched television channel; and
- determining if said identified television channel matches any listed television channel contained in said television channel list, wherein said performance index is derived from said matching determination.
847. The method of claim 844, wherein said selected one or more television channels comprises a plurality of selected television channels.
848. The method of claim 844, wherein said selected one or more television channels comprises a single selected television channel.
849. The method of claim 848, further comprising tuning said television to a television channel corresponding to said selected single television channel.
850. The method of claim 844, wherein said selected one or more television channels corresponds to the highest one or more probability values in said television channel probability distribution.
851. The method of claim 844, further comprising placing said selected one or more television channels in an order according to corresponding probability values.
852. The method of claim 844, wherein said television channel probability distribution is modified by updating said television channel probability distribution.
853. The method of claim 852, wherein said television channel probability distribution update comprises a reward-inaction update.
854. The method of claim 844, wherein said television channel probability distribution is modified by increasing a probability value.
855. The method of claim 844, wherein said television channel probability distribution is modified by adding a probability value.
856. The method of claim 855, wherein said television channel probability distribution is modified by replacing a probability value with said added probability value.
857. The method of claim 844, wherein said performance index is instantaneous.
858. The method of claim 844, wherein said performance index is cumulative.
859. The method of claim 844, wherein said television channel probability distribution is normalized.
860. The method of claim 846, wherein said television channel probability distribution is modified by updating it if said identified television channel matches said any listed television channel.
861. The method of claim 860, wherein said television channel probability distribution update comprises a reward-inaction update.
862. The method of claim 861, wherein a corresponding probability value is rewarded if said identified television channel matches said any listed television channel.
863. The method of claim 860, further comprising adding a listed television channel corresponding to said identified television channel to said television channel list if said identified television channel does not match said any listed television channel, wherein said television channel probability distribution is modified by adding a probability value corresponding to said added listed television channel to said television channel probability distribution.
864. The method of claim 863, wherein another television channel on said television channel list is replaced with said added listed television channel, and another probability value corresponding to said replaced listed television channel is replaced with said added probability value.
865. The method of claim 846, wherein said television channel probability distribution is modified by updating it only if said identified television channel matches a television channel within said selected one or more television channels.
866. The method of claim 865, wherein said television channel probability distribution update comprises a reward-inaction update.
867. The method of claim 866, wherein a corresponding probability value is rewarded if said identified television channel matches said any listed television channel.
868. The method of claim 865, wherein said television channel probability distribution is modified by increasing a corresponding probability value if said identified television channel matches a listed television channel that does not correspond to a television channel in said selected one or more television channels.
869. The method of claim 865, further comprising adding a listed television channel corresponding to said identified television channel to said television channel list if said identified television channel does not match said any listed television channel, wherein said television channel probability distribution is modified by adding a probability value corresponding to said added listed television channel to said television channel probability distribution.
870. The method of claim 869, wherein another television channel on said television channel list is replaced with said added listed television channel, and another probability value corresponding to said replaced listed television channel is replaced with said added probability value.
871. The method of claim 844, further comprising:
- generating another television channel list containing at least another plurality of listed television channels and a television channel probability distribution comprising a plurality of probability values corresponding to said other plurality of listed television channels; and
- selecting another one or more television channels from said other plurality of television channels based on said other television channel probability distribution.
872. The method of claim 871, further comprising:
- identifying a watched television channel;
- determining if said identified television channel matches any listed television channel contained in said television channel list;
- identifying another watched television channel; and
- determining if said other identified television channel matches any listed television channel contained in said other television channel list;
- wherein said performance index is derived from said matching determinations.
873. The method of claim 871, further comprising:
- identifying a watched television channel;
- determining current temporal information;
- selecting one of said television channel list and said other television channel list based on said current temporal information determination; and
- determining if said identified television channel matches any listed television channel contained in said selected television channel list, wherein said performance index is derived from said determination.
874. The method of claim 873, wherein said temporal information is a day of the week.
875. The method of claim 873, wherein said temporal information is a time of the day.
876. The method of claim 844, wherein said television channel probability distribution is updated using a learning automaton.
877. The method of claim 844, wherein said television channel probability distribution is purely frequency based.
878. The method of claim 877, wherein said television channel probability distribution is based on a moving average.
879. The method of claim 846, wherein said television channel list is one of a plurality of like television channel lists corresponding to a plurality of users, the method further comprising determining which user watched said identified television channel, wherein said list corresponds with said determined user.
880. The method of claim 879, wherein said user determination is based on the operation of one of a plurality of keys associated with said television channel control system.
881. A television channel control system having an objective of anticipating watched television channels, comprising:
- a probabilistic learning module configured for learning favorite television channels of a user in response to watched television channels by said user; and
- an intuition module configured for modifying a functionality of said probabilistic learning module based on said objective.
882. The television channel control system of claim 881, wherein said probabilistic learning module is further configured for determining a performance index indicative of a performance of said probabilistic learning module relative to said objective, and said intuition module is configured for modifying said probabilistic learning module functionality based on said performance index.
883. The television channel control system of claim 881, wherein said probabilistic learning module comprises:
- a television channel selection module configured for selecting said one or more of said plurality of favorite television channels from a plurality of television channels contained in a television channel list, said selection being based on a television channel probability distribution comprising a plurality of probability values corresponding to said plurality of listed television channels;
- an outcome evaluation module configured for determining if an identified watched television channel matches any listed television channel contained in said television channel list; and
- a probability update module, wherein said intuition module is configured for modifying said probability update module based on said matching determination.
884. The television channel control system of claim 883, wherein said one or more favorite parameter values comprises a plurality of favorite parameter values.
885. The television channel control system of claim 883, wherein said one or more favorite parameter values comprises a single favorite parameter value.
886. The television channel control system of claim 883, wherein said one or more favorite television channels correspond to one or more of the highest probability values in said television channel probability distribution.
887. The television channel control system of claim 883, wherein said television channel selection module is further configured for placing said one or more favorite television channels in an order according to corresponding probability values.
888. The television channel control system of claim 883, wherein said intuition module is configured for modifying said probability update module by directing it to update said television channel probability distribution if said identified television channel matches said any listed television channel.
889. The television channel control system of claim 888, wherein said probability update module is configured for updating said television channel probability distribution using a reward- inaction algorithm.
890. The television channel control system of claim 889, wherein said probability update module is configured for rewarding a corresponding probability value.
891. The television channel control system of claim 888, wherein said intuition module is configured for modifying said probability update module by adding a listed television channel corresponding to said identified television channel to said television channel list and adding a probability value corresponding to said added listed television channels to said television channel probability distribution if said identified television channel does not match said any listed television channel.
892. The television channel control system of claim 891, wherein another television channel on said television channel list is replaced with said added listed television channel, and another probability value corresponding to said replaced listed television channel is replaced with said added probability value.
893. The television channel control system of claim 883, wherein said intuition module is configured for modifying said probability update module by directing it to update said television channel probability distribution only if said identified television channel matches a listed television channel corresponding to one of said favorite television channels.
894. The television channel control system of claim 893, wherein said probability update module is configured for updating said television channel probability distribution using a reward- inaction algorithm.
895. The television channel control system of claim 894, wherein said probability update module is configured for rewarding a corresponding probability value.
896. The television channel control system of claim 893, wherein said intuition module is configured for modifying said probability update module by increasing a corresponding probability value if said identified television channel matches a listed television channel that does not correspond to one of said one or more favorite television channels.
897. The television channel control system of claim 893, wherein said intuition module is configured for modifying said probability update module by adding a listed television channel corresponding to said identified television channel to said television channel list and adding a probability value corresponding to said added listed television channel to said television channel probability distribution if said identified television channel does not match said any listed television channel.
898. The television channel control system of claim 897, wherein another television channel on said television channel list is replaced with said added listed television channel, and another probability value corresponding to said replaced listed television channel is replaced with said added probability value.
899. The television channel control system of claim 881, wherein said performance index is instantaneous.
900. The television channel control system of claim 881, wherein said performance index is cumulative.
901. The television channel control system of claim 881, wherein said probabilistic learning module comprises a learning automaton.
902. The television channel control system of claim 881, wherein said probabilistic learning module is purely frequency-based.
903. The television channel control system of claim 881, wherein said television channel control system is a remote control unit.
904. The television channel control system of claim 883, further comprising a favorite channel function key the operation of which prompts said television channel selection module to select said one or more television channels.
905. The television channel control system of claim 883, wherein said television channel list is one of a plurality of like television channel lists corresponding to a plurality of users, and said television channel selection module is further configured for determining which user watched said identified television channel, wherein said list corresponds with said determined user.
906. The television channel control system of claim 905, further comprising a plurality of user function keys, wherein said user determination is based on the operation of one of said plurality of user function keys.
907. A method of providing learning capability to a television channel control system, comprising:
- generating a plurality of lists respectively associated with a plurality of television channel parameter values, each of said plurality of lists containing a plurality of listed television channels with an associated television channel probability distribution comprising a plurality of probability values corresponding to said plurality of listed television channels;
- selecting a list corresponding to a television channel parameter value exhibited by a currently watched television channel; and
- in said selected list, selecting one or more listed actions from said plurality of listed actions based on said action probability distribution;
- determining a performance index indicative of a performance of said processing device relative to said objective; and
- modifying said action probability distribution based on said performance index.
908. The method of claim 907, further comprising:
- identifying a watched television channel exhibiting a television channel parameter value;
- selecting a list corresponding to said television channel parameter;
- determining if said identified watched television channel matches any listed television channel contained in said selected list, wherein said performance index is based on said matching determination.
909. The method of claim 908, wherein said selected one or more television channels comprises a plurality of selected television channels.
910. The method of claim 908, wherein said selected one or more television channels comprises a single selected television channel.
911. The method of claim 908, wherein said selected one or more television channels corresponds to the highest one or more probability values in said television channel probability distribution.
912. The method of claim 907, further comprising placing said plurality of listed television channels in an order according to corresponding probability values.
913. The method of claim 907, wherein said television channel probability distribution is modified by updating said television channel probability distribution.
914. The method of claim 913, wherein said television channel probability distribution update comprises a reward-inaction update.
915. The method of claim 907, wherein said television channel probability distribution is modified by increasing a probability value.
916. The method of claim 907, wherein said television channel probability distribution is modified by adding a probability value.
917. The method of claim 916, wherein said television channel probability distribution is modified by replacing a probability value with said added probability value.
918. The method of claim 907, wherein said performance index is instantaneous.
919. The method of claim 907, wherein said performance index is cumulative.
920. The method of claim 907, wherein said television channel probability distribution is modified by updating it if said identified television channel matches said any listed television channel.
921. The method of claim 920, wherein said television channel probability distribution update comprises a reward-inaction update.
922. The method of claim 921, wherein a corresponding probability value is rewarded if said identified television channel matches said any listed television channel.
923. The method of claim 920, further comprising adding a listed television channel corresponding to said identified television channel to said television channel list if said identified television channel does not match said any listed television channel, wherein said television channel probability distribution is modified by adding a probability value corresponding to said added listed television channel to said television channel probability distribution.
924. The method of claim 923, wherein another television channel on said television channel list is replaced with said added listed television channel, and another probability value corresponding to said replaced listed television channel is replaced with said added probability value.
925. The method of claim 907, wherein said television channel probability distribution is modified by updating it only if said identified television channel matches a television channel within said selected one or more television channels.
926. The method of claim 925, wherein said television channel probability distribution update comprises a reward-inaction update.
927. The method of claim 926, wherein a corresponding probability value is rewarded if said identified television channel matches said any listed television channel.
928. The method of claim 925, wherein said television channel probability distribution is modified by increasing a corresponding probability value if said identified television channel matches a listed television channel that does not correspond to a television channel in said selected one or more television channels.
929. The method of claim 925, further comprising adding a listed television channel corresponding to said identified television channel to said television channel list if said identified television channel does not match said any listed television channel, wherein said television channel probability distribution is modified by adding a probability value corresponding to said added listed television channel to said television channel probability distribution.
930. The method of claim 929, wherein another television channel on said television channel list is replaced with said added listed television channel, and another probability value corresponding to said replaced listed television channel is replaced with said added probability value.
931. The method of claim 907, wherein said television channel probability distribution is updated using a learning automaton.
932. The method of claim 907, wherein said television channel probability distribution is purely frequency based.
933. The method of claim 907, wherein said television channel probability distribution is based on a moving average.
934. The method of claim 907, wherein said television channel parameter comprises a switched channel number.
935. The method of claim 907, wherein said television channel parameter comprises a channel type.
936. The method of claim 907, wherein said television channel parameter comprises a channel age/gender.
937. The method of claim 907, wherein said television channel parameter comprises a channel rating.
938. A method of providing learning capability to an educational toy, comprising:
- selecting one of a plurality of toy actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of toy actions, said plurality of toy actions being associated with a plurality of different difficulty levels;
- identifying an action performed by a user;
- determining an outcome of said selected toy action relative to said identified user action; and
- updating said action probability distribution based on said outcome and said difficulty level of said selected toy action.
939. The method of claim 938, wherein said plurality of toy actions comprises a plurality of educational games.
940. The method of claim 938, wherein said plurality of toy actions comprises a plurality of educational tasks to be performed by said user.
941. The method of claim 940, wherein each of said plurality of educational tasks comprises identifying a combination of items.
942. The method of claim 938, wherein said outcome can be represented by one of two possible values.
943. The method of claim 942, wherein said two possible values are the integers “zero” and “one.”
944. The method of claim 938, wherein said outcome can be represented by one of a finite range of real numbers.
945. The method of claim 938, wherein said outcome can be represented by one of a range of continuous values.
946. The method of claim 938, wherein said identified user action is performed in response to said selected toy action.
947. The method of claim 938, wherein said outcome is determined by determining if said identified user action matches said selected toy action.
948. The method of claim 938, wherein said action probability distribution is shifted from one or more probability values corresponding to one or more toy actions associated with lesser difficulty levels to one or more probability values corresponding to one or more toy actions associated with greater difficulty levels if said outcome indicates that said identified user action is successful relative to said selected toy action.
949. The method of claim 948, wherein said one or more toy actions associated with greater difficulty levels includes a toy action associated with a difficulty level equal to or greater than said difficulty level of said selected toy action.
950. The method of claim 948, wherein said one or more toy actions associated with lesser difficulty levels includes a toy action associated with a difficulty level equal to or less than said difficulty level of said selected toy action.
951. The method of claim 938, wherein said action probability distribution is shifted from one or more probability values corresponding to one or more toy actions associated with greater difficulty levels to one or more probability values corresponding to one or more toy actions associated with lesser difficulty levels if said outcome indicates that said identified user action is unsuccessful relative to said selected toy action.
952. The method of claim 951, wherein said one or more toy actions associated with lesser difficulty levels includes a toy action associated with a difficulty level equal to or less than said difficulty level of said selected toy action.
953. The method of claim 951, wherein said one or more toy actions associated with greater difficulty levels includes a toy action associated with a difficulty level equal to or greater than said difficulty level of said selected toy action.
954. The method of claim 938, wherein said action probability distribution is normalized.
955. The method of claim 938, wherein said selected toy action corresponds to a pseudo-random selection of a probability value within said action probability distribution.
956. The method of claim 938, wherein said action probability distribution is updated using a learning automaton.
957. A method of providing learning capability to an educational toy having an objective of increasing an educational level of a user, comprising:
- selecting one of a plurality of toy actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of toy actions, said plurality of toy actions being associated with a plurality of different difficulty levels;
- identifying an action performed by said user;
- determining an outcome of said selected toy action relative to said identified user action;
- updating said action probability distribution based on said outcome; and
- modifying one or more of said toy action selection, outcome determination, and action probability distribution update based on said objective.
958. The method of claim 957, wherein said outcome is determined by determining if said identified user action matches said selected toy action.
959. The method of claim 957, further comprising determining a performance index indicative of a performance of said educational toy relative to said objective, wherein said modification is based on said performance index.
960. The method of claim 959, wherein said performance index is derived from said outcome and said difficulty level of said selected toy action.
961. The method of claim 959, wherein said performance index is cumulative.
962. The method of claim 959, wherein said performance index is instantaneous.
963. The method of claim 957, wherein said modification comprises modifying said action probability distribution update.
964. The method of claim 963, wherein said modification comprises selecting one of a predetermined plurality of learning methodologies employed by said action probability distribution update.
965. The method of claim 964, wherein a learning methodology that rewards a toy action having a difficulty level equal to or greater than said difficulty level of said selected toy action is selected if said outcome indicates that said identified user action is successful relative to said selected toy action.
966. The method of claim 964, wherein a learning methodology that penalizes a toy action having a difficulty level equal to or less than said difficulty level of said selected toy action is selected if said outcome indicates that said identified user action is successful relative to said selected toy action.
967. The method of claim 964, wherein a learning methodology that rewards a toy action having a difficulty level equal to or less than said difficulty level of said selected toy action is selected if said outcome indicates that said identified user action is unsuccessful relative to said selected toy action.
968. The method of claim 964, wherein a learning methodology that penalizes a toy action having a difficulty level equal to or greater than said difficulty level of said selected toy action is selected if said outcome indicates that said identified user action is unsuccessful relative to said selected toy action.
969. The method of claim 957, wherein said action probability distribution is normalized.
970. The method of claim 957, wherein said selected toy action corresponds to a pseudo-random selection of a probability value within said action probability distribution.
971. The method of claim 957, wherein said action probability distribution is updated using a learning automaton.
972. An educational toy having an objective of increasing an educational level of a user, comprising:
- a probabilistic learning module configured for learning a plurality of toy actions in response to a plurality of actions performed by a user; and
- an intuition module configured for modifying a functionality of said probabilistic learning module based on said objective.
973. The educational toy of claim 972, wherein said intuition module is further configured for determining a performance index indicative of a performance of said probabilistic learning module relative to said objective, and for modifying said probabilistic learning module functionality based on said performance index.
974. The educational toy of claim 972, wherein said performance index is cumulative.
975. The educational toy of claim 972, wherein said performance index is instantaneous.
976. The educational toy of claim 972, wherein said probabilistic learning module comprises:
- an action selection module configured for selecting one of a plurality of toy actions, said action selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of toy actions, said plurality of toy actions being associated with a plurality of different difficulty levels;
- an outcome evaluation module configured for determining an outcome of said selected toy action relative to said user action; and
- a probability update module configured for updating said action probability distribution based on said outcome and said difficulty level of said selected toy action.
977. The educational toy of claim 972, wherein said performance index is derived from said outcome and said difficulty level of said selected toy action.
978. The educational toy of claim 972, wherein said intuition module is configured for modifying a functionality of said probability update module based on said objective.
979. The educational toy of claim 978, wherein said intuition module is configured for selecting one of a predetermined plurality of learning methodologies employed by said probability update module.
980. The educational toy of claim 979, wherein said intuition module is configured for selecting a learning methodology that rewards a toy action having a difficulty level equal to or greater than said difficulty level of said selected toy action if said outcome indicates that said identified user action is successful relative to said selected toy action.
981. The educational toy of claim 979, wherein said intuition module is configured for selecting a learning methodology that penalizes a toy action having a difficulty level equal to or less than said difficulty level of said selected toy action is selected if said outcome indicates that said identified user action is successful relative to said selected toy action.
982. The educational toy of claim 979, wherein said intuition module is configured for selecting a learning methodology that rewards a toy action having a difficulty level equal to or less than said difficulty level of said selected toy action is selected if said outcome indicates that said identified user action is unsuccessful relative to said selected toy action.
983. The educational toy of claim 979, wherein said intuition module is configured for selecting a learning methodology that penalizes a toy action having a difficulty level equal to or greater than said difficulty level of said selected toy action is selected if said outcome indicates that said identified user action is unsuccessful relative to said selected toy action.
984. The educational toy of claim 979, wherein said probability learning module comprises a learning automaton.
985. A method of providing learning capability to a processing device, comprising:
- selecting one of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions, said plurality of processor actions being associated with a plurality of different difficulty levels;
- identifying an action performed by a user;
- determining an outcome of said selected processor action relative to said identified user action; and
- updating said action probability distribution based on said outcome and said difficulty level of said selected processor action.
986. The method of claim 985, wherein said processing device is an educational device.
987. The method of claim 986, wherein said plurality of processor actions comprises a plurality of educational games.
988. The method of claim 986, wherein said plurality of processor actions comprises a plurality of educational tasks to be performed by said user.
989. The method of claim 988, wherein each of said plurality of educational tasks comprises identifying a combination of items.
990. The method of claim 985, wherein said outcome can be represented by one of two possible values.
991. The method of claim 990, wherein said two possible values are the integers “zero” and “one.”
992. The method of claim 985, wherein said outcome can be represented by one of a finite range of real numbers.
993. The method of claim 985, wherein said outcome can be represented by one of a range of continuous values.
994. The method of claim 985, wherein said identified user action is performed in response to said selected processor action.
995. The method of claim 985, wherein said outcome is determined by determining if said identified user action matches said action.
996. The method of claim 985, wherein said action probability distribution is shifted from one or more probability values corresponding to one or more actions associated with lesser difficulty levels to one or more probability values corresponding to one or more actions associated with greater difficulty levels if said outcome indicates that said identified user action is successful relative to said selected processor action.
997. The method of claim 996, wherein said one or more actions associated with greater difficulty levels includes a action associated with a difficulty level greater than said difficulty level of said action.
998. The method of claim 996, wherein said one or more actions associated with lesser difficulty levels includes a action associated with a difficulty level less than said difficulty level of said action.
999. The method of claim 985, wherein said action probability distribution is shifted from one or more probability values corresponding to one or more actions associated with greater difficulty levels to one or more probability values corresponding to one or more actions associated with lesser difficulty levels if said outcome indicates that said identified user action is unsuccessful relative to said selected processor action.
1000. The method of claim 999, wherein said one or more actions associated with lesser difficulty levels includes a action associated with a difficulty level less than said difficulty level of said action.
1001. The method of claim 999, wherein said one or more actions associated with greater difficulty levels includes a action associated with a difficulty level greater than said difficulty level of said action.
1002. The method of claim 985, wherein said action probability distribution is normalized.
1003. The method of claim 985, wherein said selected processor action corresponds to a pseudo-random selection of a probability value within said action probability distribution.
1004. The method of claim 985, wherein said action probability distribution is updated using a learning automaton.
1005. A method of providing learning capability to a processing device having one or more objectives, comprising:
- selecting one of a plurality of processor actions based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions, said plurality of processor actions being associated with a plurality of different difficulty levels;
- identifying an action performed by said user;
- determining an outcome of said selected processor action relative to said identified user action;
- updating said action probability distribution based on said outcome; and
- modifying one or more of said action selection, outcome determination, and action probability distribution update based on said objective.
1006. The method of claim 1005, wherein said processing device is an educational device, and said one or more objectives comprises increasing the educational level of said user.
1007. The method of claim 1005, wherein said outcome is determined by determining if said identified user action matches said action.
1008. The method of claim 1005, further comprising determining a performance index indicative of a performance of said processing device relative to said objective, wherein said modification is based on said performance index.
1009. The method of claim 1008, wherein said performance index is derived from said outcome and said difficulty level of said selected processor action.
1010. The method of claim 1008, wherein said performance index is cumulative.
1011. The method of claim 1008, wherein said performance index is instantaneous.
1012. The method of claim 1005, wherein said modification comprises modifying said action probability distribution update.
1013. The method of claim 1012, wherein said modification comprises selecting one of a predetermined plurality of learning methodologies employed by said action probability distribution update.
1014. The method of claim 1013, wherein a learning methodology that rewards a action having a difficulty level equal to or greater than said difficulty level of said selected processor action is selected if said outcome indicates that said identified user action is successful relative to said selected processor action.
1015. The method of claim 1013, wherein a learning methodology that penalizes a action having a difficulty level equal to or less than said difficulty level of said selected processor action is selected if said outcome indicates that said identified user action is successful relative to said selected processor action.
1016. The method of claim 1013, wherein a learning methodology that rewards a action having a difficulty level equal to or less than said difficulty level of said selected processor action is selected if said outcome indicates that said identified user action is unsuccessful relative to said selected processor action.
1017. The method of claim 1013, wherein a learning methodology that penalizes a action having a difficulty level equal to or greater than said difficulty level of said selected processor action is selected if said outcome indicates that said identified user action is unsuccessful relative to said selected processor action.
1018. The method of claim 1005, wherein said action probability distribution is normalized.
1019. The method of claim 1005, wherein said selected processor action corresponds to a pseudo-random selection of a probability value within said action probability distribution.
1020. The method of claim 1005, wherein said action probability distribution is updated using a learning automaton.
1021. A processing device having one or more objectives, comprising:
- an action selection module configured for selecting one of a plurality of processor actions, said action selection being based on an action probability distribution comprising a plurality of probability values corresponding to said plurality of processor actions, said plurality of processor actions being associated with a plurality of different difficulty levels;
- an outcome evaluation module configured for determining an outcome of said selected processor action relative to said user action; and
- a probability update module configured for updating said action probability distribution based on said outcome and said difficulty level of said selected processor action.
1022. The processing device of claim 1021, wherein said intuition module is further configured for determining a performance index indicative of a performance of said probabilistic learning module relative to said objective, and for modifying said probability update module functionality based on said performance index.
1023. The processing device of claim 1021, further comprising an intuition module configured for modifying a functionality of said probability update module based on said one or more objectives.
1024. The processing device of claim 1023, wherein the processing device is an educational device, and said one or more objectives comprises increasing the education level of said user.
1025. The processing device of claim 1024, wherein said plurality of processor actions comprises a plurality of educational games.
1026. The processing device of claim 1024, wherein said plurality of processor actions comprises a plurality of educational tasks to be performed by said user.
1027. The processing device of claim 1026, wherein each of said plurality of educational tasks comprises identifying a combination of items.
1028. The processing device of claim 1026, wherein said intuition module is further configured for determining a performance index indicative of a performance of said probabilistic learning module relative to said objective, and for modifying said probability update module functionality based on said performance index.
1029. The processing device of claim 1028, wherein said performance index is cumulative.
1030. The processing device of claim 1028, wherein said performance index is instantaneous.
1031. The processing device of claim 1028, wherein said performance index is derived from said outcome and said difficulty level of said selected processor action.
1032. The processing device of claim 1023, wherein said intuition module is configured for selecting one of a predetermined plurality of learning methodologies employed by said probability update module.
1033. The processing device of claim 1032, wherein said intuition module is configured for selecting a learning methodology that rewards a processor action having a difficulty level equal to or greater than said difficulty level of said selected processor action if said outcome indicates that said identified user action is successful relative to said selected processor action.
1034. The processing device of claim 1032, wherein said intuition module is configured for selecting a learning methodology that penalizes a processor action having a difficulty level equal to or less than said difficulty level of said selected processor action is selected if said outcome indicates that said identified user action is successful relative to said selected processor action.
1035. The processing device of claim 1032, wherein said intuition module is configured for selecting a learning methodology that rewards a processor action having a difficulty level equal to or less than said difficulty level of said selected processor action is selected if said outcome indicates that said identified user action is unsuccessful relative to said selected processor action.
1036. The processing device of claim 1032, wherein said intuition module is configured for selecting a learning methodology that penalizes a processor action having a difficulty level equal to or greater than said difficulty level of said selected processor action is selected if said outcome indicates that said identified user action is unsuccessful relative to said selected processor action.
1037. The processing device of claim 1021, wherein said probability update module is configured for using a learning automaton to update said action probability distribution.
Type: Application
Filed: Aug 30, 2002
Publication Date: Aug 7, 2003
Patent Grant number: 7483867
Applicant: Intuitive Intelligence, Inc. (Costa Mesa, CA)
Inventors: Arif M. Ansari (Los Angeles, CA), Yusuf Sulaiman M. Shiek Ansari (Costa Mesa, CA)
Application Number: 10231875
International Classification: G06F015/18; G06E001/00; G06E003/00; G06N003/02; G06G007/00; G06N003/08;
