US20070260567A1 - Real time context learning by software agents - Google Patents
Real time context learning by software agents Download PDFInfo
- Publication number
- US20070260567A1 US20070260567A1 US10/885,495 US88549504A US2007260567A1 US 20070260567 A1 US20070260567 A1 US 20070260567A1 US 88549504 A US88549504 A US 88549504A US 2007260567 A1 US2007260567 A1 US 2007260567A1
- Authority
- US
- United States
- Prior art keywords
- player
- response
- behavior
- context
- player characters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/60—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- A63F13/67—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/60—Methods for processing data by generating or executing the game program
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/60—Methods for processing data by generating or executing the game program
- A63F2300/6027—Methods for processing data by generating or executing the game program using adaptive systems learning from user actions, e.g. for skill level adjustment
Definitions
- the invention relates to context learning by software agents, such as for example might be used by embedded game agents in a gaming system, such as for example during play of the game.
- a computer simulation especially in a computer game that attempts to present a fantasy world (i.e., to create and simulate a variant world in which there are at least some fantasy or fictional aspects), there might be numerous agents within the game, each of which has independent actions they can perform.
- One set of those agents are controlled by players (herein sometimes called “player characters,” and herein sometimes called “human users”), while other sets of those agents are controlled by the game software (herein sometimes called “non-player characters”). It is also possible that the selection of which of those agents are player characters and the selection of which of those agents are non-player characters might change from time to time as the game progresses, perhaps as the player chooses to control different characters.
- player characters are entertaining, intelligent, realistic, or robust, it would be advantageous for non-player characters to exhibit at least some of these qualities, or to exhibit these qualities to at least some of the degree that player characters do.
- player characters exhibit one or more elements of oddity or quirkiness, it would be advantageous for non-player characters to exhibit at least some of those elements, or similar elements of oddity or quirkiness, or other aspects of the players' personality, as expressed through those players' playing style.
- non-player characters could learn in real time (that is, could modify their behavior and their response to various stimuli during actual play) from at least some of these possible sources of potential learning, including learning from player characters, from other non-player characters, or from their interaction with the game environment. More specifically, it would also be advantageous if each distinct player character can “teach” (for example, serve to model behavior for) its own set of non-player characters. Those sets of non-player characters might overlap, such as when the non-player characters learn the playing style of the player characters generally, or might be distinct, such as when the non-player characters are distributed into teams, each team of which learns the playing style of a specific player character during a specific time frame.
- non-player characters could learn from a particular player character, that player might see their tactics and strategies, and possibly their personality, replicated and possibly amplified by a set of non-player characters (whether friendly non-player characters or enemy non-player characters).
- a set of non-player characters whether friendly non-player characters or enemy non-player characters.
- distinct teams of non-player characters could learn from multiple distinct player characters, those distinct player characters might each develop their own team of non-player characters, each team with its own particularized behavior and each team with its own leadership and cooperative strategies, in response to the ability or initiative of those player characters in training or re-training their respective teams.
- non-player characters In a first set of known systems, non-player characters have a set of pre-programmed behavior, which they follow no matter who the player is, how the player acts, or how the game progresses.
- the set of possible actions can be pre-programmed by the game designer, with a known set of actions to be performed in known possible circumstances. This is sometimes called a “script.”
- a script might include random or pseudorandom selection of actions, in response to parameters selected by the game designer.
- the parameters selected by the game designer can be adjusted to cause the global apparent behavior of the non-player character to change with time.
- a non-player character might maintain a model of how to behave (sometimes herein called a “character model of behavior” or a “behavior model”), and attempt to update that model in response to aspects of the game environment.
- a model of how to behave sometimes herein called a “character model of behavior” or a “behavior model”
- a “model” refers to a set of internal state indicative of information or techniques possibly learned by the non-player character; for example, if the non-player character were using a neural network to learn behavior, that model would include the connectivity and weights assigned to nodes and edges in that neural network, as well as the particular topology of that neural network, and methods of using the neural network to generate responses to queries.)
- a non-player character might update that model in response to actions taken by a player character (for the purpose attempting to imitate that player character), or might update that model in response to experiences in the game environment (for the purpose of attempting to learn from its mistakes).
- a further drawback of known methods is that they generally involve substantial knowledge about both (1) machine learning techniques, and (2) sets of derived features likely to be valuable for machine learning, each of which has the effect, and in combination which have the effect, of involving substantial effort, time, and possibly monetary resources for the game designer to implement. It would be advantageous to provide a system in which non-player characters, or other actors, can learn from player characters, without requiring substantial investment of effort, money, or time by the game designer.
- the invention provides a method and system capable of providing dynamic learning for software agents in a simulation, such as for example non-player characters in a computer game.
- the method and system provide software agents with one or more learners, that, is, software elements capable of learning from examples, capable of receiving requests from non-player characters and capable of responding to those requests with responses from which the non-player characters can readily derive one or more actions to take.
- the learner gleans behavior from actions taken in context by a player character. This has the effect that when a non-player character queries that learner for a next action, the learner can provide a next action similar to one that might be taken by the player character (if the player character were directing the actions of that non-player character).
- the invention includes a method and system for adding a set of learners to the game structure, and a computer game with those learners added (that is, a method of playing a computer game, and a system for playing that computer game).
- the method and system include a set of derived features, responsive to raw features.
- Raw features include raw data that indicate information about a game environment, such as for example locations of game objects, distances between game objects, a direction of a most valuable visible object, measures of wounded status of player characters and non-player characters, measures of damage to non-living objects such as walls, and the like.
- Derived features might be relatively simply related to raw features, such as for example an indicator of a direction to move to get to a nearest enemy or treasure, or a measure of a distance thereto.
- Derived features might be more complex, such as for example (1) an expert system in which at least some state information is retained from time to time, or (2) a set of scripted instructions, or (3) a state machine operated in response to other raw features or derived features. Derived features might be still more complex, such as for example determined in response to a query to, and a response from, a learner. The set of such raw features and derived features, in combination, collectively form a context to which a learner is responsive. This has the effect that learners might be responsive to (more basic) learners, or to a set of results of (expert) state machines, or to a set of calculated derived features, or to a set of raw features, or to some combination or conjunction thereof.
- the learner includes one or more machine learning techniques, including for example, decision trees, hidden Markov models, Bayesian learning techniques, and other alternatives.
- machine learning techniques are known in the arts of artificial intelligence and machine learning. Among the many alternatives include techniques related to evolution strategies, genetic algorithms, genetic programming, multidimensional clustering, neural networks, and weighted majority techniques.
- the learner might include a weighted average of a set of relatively simpler elements, updated in real time during actual game play using an exponential gradient technique, or some other machine learning technique.
- multiple actions might be separated into control by different learners, or where alternatively appropriate, multiple actions might be consolidated into control by one earner.
- FIG. 1 shows a block diagram of a representation of a game system including context learning by software agents.
- FIG. 2 shows a block diagram of an example of context learning by software agents.
- FIG. 3 shows a process flow diagram of a method of operating in a system for context learning by software agents.
- FIG. 1 shows a block diagram of a representation of a game system including context learning by software agents.
- a game 100 on a game system 105 includes a game engine 110 , which in turn includes game code 120 , game-state 130 , and one or more software agents 140 .
- the game code 120 includes a set of software processes for controlling the presentation of a computer game.
- Game code 120 typically is provided or coded by one or more game designers and programmers.
- the game code preferably includes program structures such as data structures, data types, function entry points, function pointers, object data types, object methods, and the like. Some, all, none or different structure might be present depending on the way in which game 100 is designed.
- game code 120 is shown in FIG. 1 as a separate box, the other elements of the game engine 110 typically are formed from portions of game code 120 .
- the game-state 130 includes a set of state information, preferably used during operation of the game engine 110 . Although the game-state 130 is shown in FIG. 1 as a separate box, the game-state can encompass variables and data structures used by the other element of the game engine 110 .
- the one or more software agents 140 collectively include one or more player characters 141 and non-player characters 142 .
- a player character 141 includes a video object controlled by a human user 143 (or a team of such human users 143 ).
- a human user 143 or a team of such human users 143 .
- “Lara Croft” of the “Tomb Raider” Series
- “Jak” of the “Jak and Daxter” game
- a non-player character 142 includes a video object controlled by the system running the application software.
- the four roaming ghosts (“Blinky,” “Pinky,” “Inky,” and “Clyde”) in the classic “PacMan” game are examples of non-player characters 142 .
- game controllers or other input devices to affect the actions of at least some of the non-player characters 142 , such as for example the second “Ice Climbers” character in “Super Smash Brothers Melee” (for the Nintendo GameCube).
- the program code for the game engine 110 includes elements capable of specifying relevant information to other elements in the game 100 .
- game engine 110 might include program structures having information relating to raw features 150 .
- the raw features 150 indicate important information about the game 100 , such as for example a location of a game objects, a point value of a game object, a measure of a wounded status of a player character 141 or non-player character 142 , and the like.
- the program structures for raw features 150 might include C language code (or C++ language code, or code in another language) describing those data structures and how they can be referenced (such as for example a word offset within a data structure).
- game engine 110 might include references to object methods capable of computing raw features 150 , such as for example a distance between game objects, a direction of the most valuable object visible, and the like.
- the program structures for raw features 150 might include C language code (or C++ language code, or code in another language) describing locations of those object methods or instructions for performing them, and how they can be referenced (such as for example a function-call address).
- a raw feature 150 might include one or more of the following data:
- raw features 150 such as for example a location of a game objects, a point value of a game object, a measure of a wounded status of a player character 141 or non-player character 142 , and the like.
- the derived features 160 include a set of features derived in response to the raw features 150 , such as for example by calculating the values of those derived features 160 using the values of the raw features 150 .
- a derived feature 160 might include a weighted sum of a set of raw features 150 , a function calculation in response to a set of raw features 150 , and the like.
- a derived feature 160 might include a “best path” from a non-player character 142 to a target object, with the distance and direction for that best path pre-computed for each point thereon.
- a derived feature 160 might be determined using one or more predefined calculations or operations performed on one or more raw features, raw features and derived features, or both raw features and derived 11 features. Some of the following examples are based on such predefined calculations or operations.
- a derived feature 160 might also or instead be determined using one or more learned or derived calculations or operations performed on one or more raw features, raw features and derived features, or both raw features and derived features. Some of the following examples are based on such predefined calculations or operations.
- a derived feature 160 might include one or more of the following data:
- One aspect of the invention is the provision of a selected set of derived features needed for the learner 165 to be able to learn effectively.
- the inventors have found that it is non-obvious and unexpectedly useful to provide such a selected set of derived features.
- Learner 165 is capable of providing dynamic learning for software agents in game 100 .
- Learner 165 is capable of learning from examples, capable of receiving requests from non-player characters and capable of responding to those requests with responses from which the non-player characters can readily derive one or more actions to take.
- the learner 165 gleans behavior from actions taken in context by a player character. This has the effect that when a non-player character queries that learner for a next action, the learner can provide a next action similar to one that might be taken by the player character (if the player character were directing the actions of that non-player character).
- learner 165 makes predictions and generates responses to queries for given contexts based on learned behavior.
- Learner 165 can also be referred to as a simulation engine.
- learner 165 includes learning element(s) 170 , behavior model(s) 180 and decision model(s) 190 .
- the invention is not limited to learners constructed as shown in FIG. 1 .
- Inputs to leaning element(s) 170 include contexts 200 and training signal 210 .
- Each context 200 includes a set of values for the derived features 160 and possibly for the raw features 150 .
- the number of possible contexts might be quite large. For example, not intended to be limiting in any way, if there are several hundred raw features 150 , each of which might take on one of ten possible values, and there are several hundred derived features 160 determined in response to those raw features 150 , the number of possible contexts 170 might be larger than 10 raised to the power of hundreds.
- Training signal 210 includes decisions made by one or more of software agents 140 . Training signal 210 can also include comparisons between actions predicted by learner 165 and actions actually taken by a software agent. Thus, learning element(s) 170 are trained based on what decisions one or more software agents 140 make in various contexts.
- learning element(s) 170 are the portions of learner 165 that are primarily responsible for generating signals that result in machine learning. In particular, learning element(s) 170 generate updates 175 for behavior model(s) 180 .
- a behavior model 180 attempts to model behaviors exhibited by software agents 140 in various contexts 200 .
- Behavior model(s) 180 preferably generate predictions based on context information. The predictions from behavior model(s) 180 attempt to approximate what a player character or trained non-player character would do in the given context.
- a behavior model 180 includes a set of characteristics that (at least partially) predict the interaction of a non-player character 142 in various contexts during game-play. These characteristic could take the form of coefficients, weights, or other values
- the context information for behavior model(s) 180 is contexts 200 from derived features 160 .
- the same or modified context information can be supplied to the behavior model(s) 180 from decision model(s) 190 .
- the decision model(s) could ask for a prediction about a context that might occur based on a current context but that is not actually occurring.
- a decision model 190 uses one or more behavior model(s) 180 to respond to queries 220 from game engine 110 , given contexts 200 . Accordingly, a decision model 190 is responsive to a set of contexts 200 , and is capable of providing a decision for an action by a non-player character 142 . For example, a decision model 190 might provide a decision for a non-player character 142 regarding whether, in a particular context, fight or flight is better for that non-player character 142 .
- the decision model(s) 190 can be viewed as wrapping the behavior model(s) 180 , in the sense that a decision model takes a prediction from one or more behavior model(s) and can choose to adapt or modify the prediction to answer a query.
- the decision model(s) and behavior model(s) can be viewed as separate entities, in the sense that the behavior model(s) are more involved in updating and learning, whereas the decision model(s) are more involved in predicting behavior and responding to queries.
- the learner 165 embodies one or more machine learning techniques. Accordingly, the learner 165 would be ultimately responsive to the raw features 150 and derived features 160 in each of contexts 200 , and thus responsive to each context 200 as it occurs, to provide a response to a query from the game engine 110 .
- Multiple learning elements 170 , behavior models 180 and decision models 190 might be combined into one or more learners 165 , each for responding to queries in different situations. These multiple learners, learning elements, and/or models might control distinct aspects of the actions taken by a non-player character 142 (in a particular context 170 ). For one example, not intended to be limiting in any way, a first learning element 180 might determine the fight-or-flight response (as described above) for the non-player character 142 , a second learner 165 might determine what direction and how best to conduct flight when that response is chosen by the first learner 165 , and a third learner 165 might determine how to conduct combat when that response is chosen by the first learner 165 .
- a learner 165 might be responsive to include one or more of the following queries and provide one more of the following responses:
- the game engine 110 compares those responses with the actions of a target whose behavior the non-player character 142 seeks to imitate.
- the target might include a particular player character 141 , with the effect that the decisions made by the learner 165 (as requested by the game engine 110 ) are compared with actual decisions made by the player character 141 .
- learner 165 learns by having learning element(s) 170 update behavior model(s) 180 , which in turn affect decision model(s) 190 .
- the invention is not limited to this particular arrangement. Other designs of elements and modules that result in machine learning or other forms of artificial intelligence can be used with the invention. In those other designs, each of learning element(s) 170 , behavior model(s) 180 , and decision model(s) 190 might or might not be present, might have different responsibilities, or might be replaced with one or more other elements or modules.
- a human player 143 generally makes better decisions the next time they encounter a situation based on the success of prior decisions and their outcomes.
- Simple Artificial Intelligence in computer games often does not learn because it generally retains no prior knowledge of past experiences, and when it does, it does not learn from that experience so as to act differently in that context a next time.
- Enabling a game that learns to interact more like human beings keeps the game fresh for the human user 143 .
- “PacMan” was a very popular game in the early eighties. After publication of known “play patterns” for its non-player characters 142 , anyone who knew the patterns could potentially score an unlimited number of points. Essentially, the game was beaten and gamers looked to other games that presented a challenge.
- the learner 165 enables techniques whereby a game designer or programmer unskilled in artificial intelligence can take advantage of more powerful learning techniques, overcoming these short-falls of prior games that do not learn or that learn through simple AI techniques implemented by game designers and programmers.
- decisions made by the learner 165 are compared with actual decisions made by the player character 141 .
- Training signal 210 can represent the actual decisions, a comparison of responses 230 to those decisions, or some other measure of actual decisions.
- Learning element 170 of learner 165 can use this training signal to update behavior model 180 .
- learner 165 can implement a predictive technique in which learner 165 attempts to predict the training signal 210 for each context 200 .
- learner 165 is sufficiently good at predicting that training signal, it might be said that the learner 165 “learned to imitate” a target player character 141 .
- the behavior model(s) 180 of learner 165 might use weighted averages, median calculations, or other statistical measures to make predictions.
- Learning element(s) 170 could update the coefficients or other values used to implement the statistical measures based on contexts 200 and training signal 210 .
- the weights or coefficients for a behavior model 180 are real-valued coefficients or other values as precise as available using machine arithmetic.
- the coefficients or statistical measure there is no particular requirement for selecting the coefficients or statistical measure from any particular number field.
- learner 165 can be used by learner 165 instead of or in addition to the exponential gradient technique.
- examples of such other techniques include but are not limited to a Bayesian classifier, a neural network, an expert system, a hidden Markov model, a partially observable Markov decision process model, etc.
- the game engine 110 might provide a human user 143 with access to attributes within a non-player character 142 generated by the learner 165 , such as for example to correct any flaws, or such as for example to add flaws, such as when the non-player character 142 plays the game too well and is hard to beat.
- non-player characters 142 might start with any one of the following:
- non-player characters 142 might be coupled to a training signal for learning, such as any one of the following:
- non-player characters 142 might be coupled to a learner 165 for learning, such as any one of the following:
- a production run of a game 100 or application within the game 100 might include at least some non-player characters 142 for each degree of development in their behavior model 180 .
- behavior models for the non-player character 142 might be saved for later use, either in the same game 100 or in another copy of that game 100 (that is, behavior models might be communicated or carried on physical media from one game 100 to another game 100 ).
- behavior models might be communicated or carried on physical media from one game 100 to another game 100 .
- the storage element 240 includes a device for storing electronic data including but not limited to a hard disk drive, an optical drive, a silicon chip storage element (smart media card, video game external storage card, random access memory). Code and data for the game 100 can be saved to the storage 240 .
- data that is generated by the learner 165 might be saved and loaded, either on demand, or in response to particular inputs (such as for example a game controller button).
- This data might be used as an initial behavior model 180 , that is, pre-training, or might be used as a static behavior model 180 , that is, post-training.
- the behavior model 180 might be incorporated into one or more software agents 140 , such as for example non-player characters 142 .
- new non-player characters might be generated with their initial behavior model 180 at least partially specified by the human player 143 .
- the invention enables a multitude of new modalities, mechanisms and options (collectively referred to as techniques below) for game designers. Examples, not intended to be limiting in any way, are described below. Application of the invention to these modalities, mechanisms and options would not require further invention or undue experimentation. These applications include techniques that support learning behavior either during actual game play, in a batch mode, or a combination thereof.
- the invention enables a technique that supports training a new player character by providing hints or indicators of what other player characters (or scripts designed for non-player characters would do in a similar context. Application of the invention to these techniques would not require further invention or undue experimentation.
- Another technique enabled by the invention includes “deal reckoning,” in which a game (or other virtual simulation) might be performed on more than one computing device.
- a game might be played using two logically or physically remote computing devices A (including controlling a first player character X at computing device A) and B (including controlling a second player character Y at computing device B).
- Each computing device might use a prediction function predictLocation (character, futureTime) to predict a location of that selected character at that selected futureTime.
- computing device A knows that character X is in a substantially different location, that is, computing device A's computation of XA ⁇ actualLocation (X, t 1 )>some minimum threshold tolerance, computing device A knows it needs to send information to computing device B to update the latter's value of actualLocation (X, t 1 ).
- the predictor predictLocation ( ) can be improved by learning in context, either at computing device A, computing device B, or both.
- computing devices A and B might need (but alternatively might dispense with) sending information to update each others' decision model for predictLocation ( ), but the inventors expect that total communication traffic between by computing devices A and B would be substantially reduced.
- predictLocation ( ) has a clear training signal, as whenever it is wrong by more than some minimum threshold tolerance, it is corrected to provide the actual location value.
- FIG. 2 shows a block diagram of an example of context learning by software agents. The invention is not limited in any way to the details of this example.
- Raw features 150 are shown in FIG. 2 as including various values and indications for game code 120 .
- a learner according to the invention generates the derived features 160 from the raw features 150 and possibly from other derived features 160 .
- the game code is for a simulator such as a flight simulator.
- the various raw features 150 relate to information about the flight simulation.
- the particular raw features shown in FIG. 2 are meant to be illustrative only; actual code for a flight simulator might use all, some, none, or different raw features.
- the derived features 160 in FIG. 2 represent relationships between some of raw features 150 . Derived features 160 according to the invention are not limited to such relationships. Furthermore, the particular relationships shown in FIG. 2 are meant to be illustrative only; actual derived features for a flight simulator might use all, some, none, or different derived features.
- the learner In order to generate the derived features, the learner must know where in the game code 120 to find those features. This information preferably is extracted from the code, preferably based on information supplied by the game's developer or designer, and based on templates supplied for this purpose.
- the derived features 160 form one or more contexts 200 . These contexts are supplied to learning element(s) 170 and behavior element(s) 180 , where they are used to generate predictions 185 .
- the contexts also are supplied to decision model(s) 1909 , where they are used in conjunction with predictions 185 to respond to queries 220 with responses 230 . These queries and responses are used by game code 120 to incorporate the learning capabilities of the invention into the code.
- the game code supplies training signals 210 to learning element(s) 170 , which use the signals to update behavior element(s) 180 .
- the behavior element(s) in turn affect the operation of decision model(s) 190 .
- FIG. 3 shows a process flow diagram of a method of operating in a system for context learning by software agents.
- a method 300 is performed by the game 100 . Although the method 300 is described serially, the flow points and steps of the method 300 can be performed by separate elements in conjunction or in parallel, whether asynchronously or synchronously, in a pipelined manner, or otherwise. There is no particular requirement that the method 300 must be performed in the same order in which this description lists flow points or steps, except where explicitly so indicated.
- the method 300 is ready to begin associating and updating behavior model(s) 180 with software agents 140 .
- the game 100 is activated, and the method 300 begins to operate in conjunction or combination with the game 100 .
- the method 300 determines if there should be any non-player characters 142 with pre-loaded behavioral and decision model(s). If so, the method 300 performs the next step (the step 304 ), and continues with the step thereafter (the step 305 ). If not, the method 300 skips the next step (the step 304 ), and continues with the step thereafter (the step 305 ).
- the method 300 receives one or more behavioral models 180 and decision models 190 from the storage element 240 , and incorporates or associates those models to non-player characters 142 as determined in the previous step (the step 303 ).
- the method 300 determines if any non-player characters 142 (or other software agents 140 ) should be coupled to a learner 165 for learning. If so, the method performs the next step (the step 306 ), and continues with the step thereafter (the step 307 ). If not, the method 300 skips the next step (the step 306 ), and continues with the step thereafter (the step 307 ).
- the method 300 couples one or more non-player characters 142 to learners 165 , as determined in the previous step (the step 305 ).
- the method 300 updates the behavior models 180 , for example using learning elements 170 acting in response to contexts 200 , which in turn are based on raw features 150 and derived features 160 , and in response to training signal(s) 210 .
- the contexts might include a set of changes from earlier recent contexts.
- the context of the invention there is no particular requirement for any special format for this information. Either all of contexts 200 , or parts thereof, or a set of differences, or some of them, might be used.
- the training signals could be actions actually taken by a player or non-player character, a comparison of actions taken with earlier responses from leaner(s) 165 or predictions by behavior model(s) 180 , or some other type of feedback.
- Learning elements 170 update behavior model(s) 180 responsive to these training signals, for example by updating coefficients, weights, or other values used by the behavior model(s).
- the method 300 receives queries, for example for what action a non-player character should take, from one or more of the software agents 140 .
- the behavior model(s) 180 make predictions about what action a player character or some other character would take in the current context.
- the decision model(s) 190 make decisions responsive to the predictions.
- a response to the query is sent to the querying software agent responsive to the decisions.
- steps 310 and 311 might be skipped in a training mode.
- step 307 might be skipped.
- Other variations are possible.
- steps 307 to 311 that are performed preferably are repeatedly executed so as to facilitate learning by the software agents.
- contexts 200 change, permitting the agents to learn behaviors for different contexts. This continues until the game ends in steps 312 and 313 .
- the invention enables a multitude of new modalities, mechanisms and options (collectively referred to as techniques below) for game designers. Examples, not intended to be limiting in any way, are described below. Application of the invention to these modalities, mechanisms and options would not require further invention or undue experimentation.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
Description
- 1. Field of the Invention
- The invention relates to context learning by software agents, such as for example might be used by embedded game agents in a gaming system, such as for example during play of the game.
- 2. Related Art
- In a computer simulation, especially in a computer game that attempts to present a fantasy world (i.e., to create and simulate a variant world in which there are at least some fantasy or fictional aspects), there might be numerous agents within the game, each of which has independent actions they can perform. One set of those agents are controlled by players (herein sometimes called “player characters,” and herein sometimes called “human users”), while other sets of those agents are controlled by the game software (herein sometimes called “non-player characters”). It is also possible that the selection of which of those agents are player characters and the selection of which of those agents are non-player characters might change from time to time as the game progresses, perhaps as the player chooses to control different characters.
- One problem with such systems is that it is difficult to make non-player characters learn about the game in real-time (that is, as it is played) or in response to a exemplar such as a player character (that is, in response to choices made by a human user). There are several possible sources of potential learning, including for example, from current player characters, from records of play of player characters, from the non-player characters' own experiences, and from records of play of other non-player characters. It would be advantageous if non-player characters might become more like the player characters in their behavior, as this would lead to a richer and possibly more challenging game environment. For a first example, if player characters are imaginative, intelligent, realistic, or robust, it would be advantageous for non-player characters to exhibit at least some of these qualities, or to exhibit these qualities to at least some of the degree that player characters do. For a second example, if player characters exhibit one or more elements of oddity or quirkiness, it would be advantageous for non-player characters to exhibit at least some of those elements, or similar elements of oddity or quirkiness, or other aspects of the players' personality, as expressed through those players' playing style.
- It would also be advantageous if non-player characters could learn in real time (that is, could modify their behavior and their response to various stimuli during actual play) from at least some of these possible sources of potential learning, including learning from player characters, from other non-player characters, or from their interaction with the game environment. More specifically, it would also be advantageous if each distinct player character can “teach” (for example, serve to model behavior for) its own set of non-player characters. Those sets of non-player characters might overlap, such as when the non-player characters learn the playing style of the player characters generally, or might be distinct, such as when the non-player characters are distributed into teams, each team of which learns the playing style of a specific player character during a specific time frame.
- For a first example, if non-player characters could learn from a particular player character, that player might see their tactics and strategies, and possibly their personality, replicated and possibly amplified by a set of non-player characters (whether friendly non-player characters or enemy non-player characters). For a second example, if distinct teams of non-player characters could learn from multiple distinct player characters, those distinct player characters might each develop their own team of non-player characters, each team with its own particularized behavior and each team with its own leadership and cooperative strategies, in response to the ability or initiative of those player characters in training or re-training their respective teams.
- In a first set of known systems, non-player characters have a set of pre-programmed behavior, which they follow no matter who the player is, how the player acts, or how the game progresses. (1) The set of possible actions can be pre-programmed by the game designer, with a known set of actions to be performed in known possible circumstances. This is sometimes called a “script.” (2) More generally, a script might include random or pseudorandom selection of actions, in response to parameters selected by the game designer. (3) More generally, the parameters selected by the game designer can be adjusted to cause the global apparent behavior of the non-player character to change with time.
- While these known methods generally achieve the goal of providing non-player characters with actions to take in each context expressed within the game, they have drawbacks. In general, the non-player characters do not learn, in real-time, to behave like a player character, and with the result that they typically remain less like player characters and with the effect that they typically remain less imaginative, intelligent, realistic, and robust than nearly all player characters. Lack of ability to learn has the effect that non-player characters are clearly inferior to, and do not share any salient characteristics of, the personality of any individual user.
- In a second set of systems, known in some academic circles, a non-player character might maintain a model of how to behave (sometimes herein called a “character model of behavior” or a “behavior model”), and attempt to update that model in response to aspects of the game environment. (In this context, a “model” refers to a set of internal state indicative of information or techniques possibly learned by the non-player character; for example, if the non-player character were using a neural network to learn behavior, that model would include the connectivity and weights assigned to nodes and edges in that neural network, as well as the particular topology of that neural network, and methods of using the neural network to generate responses to queries.) For example, a non-player character might update that model in response to actions taken by a player character (for the purpose attempting to imitate that player character), or might update that model in response to experiences in the game environment (for the purpose of attempting to learn from its mistakes).
- These known systems have two substantial drawbacks if they are to be used at run-time with known game systems: (1) they consume far too much in the way of computing and memory resources, in response to the very large number of possible contexts, and (2) models being learned in real-time are typically not sufficiently robust to be created and used in the course of actual game play.
- A further drawback of known methods is that they generally involve substantial knowledge about both (1) machine learning techniques, and (2) sets of derived features likely to be valuable for machine learning, each of which has the effect, and in combination which have the effect, of involving substantial effort, time, and possibly monetary resources for the game designer to implement. It would be advantageous to provide a system in which non-player characters, or other actors, can learn from player characters, without requiring substantial investment of effort, money, or time by the game designer.
- Accordingly, it would be advantageous to provide a method and system not subject to drawbacks of known systems.
- The invention provides a method and system capable of providing dynamic learning for software agents in a simulation, such as for example non-player characters in a computer game. The method and system provide software agents with one or more learners, that, is, software elements capable of learning from examples, capable of receiving requests from non-player characters and capable of responding to those requests with responses from which the non-player characters can readily derive one or more actions to take. In one embodiment, the learner gleans behavior from actions taken in context by a player character. This has the effect that when a non-player character queries that learner for a next action, the learner can provide a next action similar to one that might be taken by the player character (if the player character were directing the actions of that non-player character).
- In one embodiment, the invention includes a method and system for adding a set of learners to the game structure, and a computer game with those learners added (that is, a method of playing a computer game, and a system for playing that computer game).
- In one embodiment, the method and system include a set of derived features, responsive to raw features. Raw features include raw data that indicate information about a game environment, such as for example locations of game objects, distances between game objects, a direction of a most valuable visible object, measures of wounded status of player characters and non-player characters, measures of damage to non-living objects such as walls, and the like. Derived features might be relatively simply related to raw features, such as for example an indicator of a direction to move to get to a nearest enemy or treasure, or a measure of a distance thereto. Derived features might be more complex, such as for example (1) an expert system in which at least some state information is retained from time to time, or (2) a set of scripted instructions, or (3) a state machine operated in response to other raw features or derived features. Derived features might be still more complex, such as for example determined in response to a query to, and a response from, a learner. The set of such raw features and derived features, in combination, collectively form a context to which a learner is responsive. This has the effect that learners might be responsive to (more basic) learners, or to a set of results of (expert) state machines, or to a set of calculated derived features, or to a set of raw features, or to some combination or conjunction thereof.
- In one embodiment, the learner includes one or more machine learning techniques, including for example, decision trees, hidden Markov models, Bayesian learning techniques, and other alternatives. Several machine learning techniques are known in the arts of artificial intelligence and machine learning. Among the many alternatives include techniques related to evolution strategies, genetic algorithms, genetic programming, multidimensional clustering, neural networks, and weighted majority techniques. In addition, the learner might include a weighted average of a set of relatively simpler elements, updated in real time during actual game play using an exponential gradient technique, or some other machine learning technique.
- In one embodiment, there might be one or more learners associated with actions to be taken by selected software agents. For example, for a non-player character in a computer game, there might be separate learners for direction of movement and style of movement, and a separate learner for combat actions. Similarly, in a computer game in which such movement is possible, there might be a separate learner for direction of movement, and a separate learner for hopping, jumping, somersaulting, or otherwise indulging in acrobatics. Where appropriate, multiple actions might be separated into control by different learners, or where alternatively appropriate, multiple actions might be consolidated into control by one earner.
- After reading this application, those skilled in the art would recognize that the systems described herein provide an enabling technology, with the effect that heretofore advantageous features can be provided that heretofore were substantially infeasible.
-
FIG. 1 shows a block diagram of a representation of a game system including context learning by software agents. -
FIG. 2 shows a block diagram of an example of context learning by software agents. -
FIG. 3 shows a process flow diagram of a method of operating in a system for context learning by software agents. - Preferred embodiments of the invention are described herein, including preferred device coupling, device functionality, and process steps. After reading this application, those skilled in the art would realize that embodiments of the invention might be implemented using a variety of other techniques not specifically described herein, without undue experimentation or further invention, and that such other techniques would be within the scope and spirit of the invention.
- Definitions
- The general meaning of each of these terms is intended to be illustrative and in no way limiting.
-
- The phrase “game system” generally describes any computing device or other electronic system on which games might be played. Game systems include those units that are standalone consoles, portable devices, and general purpose computers capable of supporting game software (for example laptop and desktop computers). Known examples of standalone consoles include the Sony “PlayStation 2,” the Nintendo “Game Cube,” and the Microsoft “X-Box.”
- The phrase “computer game” or “game” generally describes software or instructions, or rules describing an interactive system, performed by a computing device or other electronic system and intended for use by a user. Computer games include character “fighting” games, hand-eye coordination or “shooter” games, mazes and puzzles, problem-solving games, racing or piloting games, role-playing games, simulation games, and other games. In the context of the invention, there is no particular requirement that computer games are designed to be played by children (games with adult-oriented content are known), or that computer games are designed to be played for fun (games with educational intent are known), or that computer games have a “winning” or “losing” state (games with simulation effect, such as for example “Sim City 3000,” are known). This concept is intended to broadly encompass any system in which a user interacts with a computing device, or with other users, according to rules (typically described and implemented by the software or instructions), even if those rules might be changed.
- The phrases “game actors,” “game agents,” “software agents,” and “game characters” (including both “player” characters and “non-player” characters) generally describe active software within the computer game, capable of making decisions (to anthropomorphize a bit) in response to contexts in which that active software responds. In the case of game characters, a player character would be primarily controlled by a user of the game (although there might be times when the game limits decisions for that character or imposes decisions on that character) and a non-player character would be primarily controlled by the game itself (although there might be times when the game allows a user to take all or partial control for that character).
- The phrases “game environment,” “game experiences,” “game play,” “game-state” or “state information,” and “simulation” generally describe the state of the game system and the computer game, including the raw features, derived features, and contexts to which software agents respond.
- The phrases “machine learning,” “artificial intelligence,” and the like interchangeably refer to processes and techniques by which computers or other devices emulate learning and learned behavior, for example to emulate or to predict actions of humans or other actors. (Technically, machine learning is a subset of artificial intelligence; however, the terms are generally used in a broader and more interchange sense in this application.)
- The phrases “context,” “actions taken in context,” generally describe a set of features, which commonly taken together, indicate a state of the game system. A context can be generically thought of as a point (or a set of points, such as for example a cluster) selected from Rn, where n is the number of possible features and R represents the set of real numbers; that is, each context describes a set of possible values for all variables available in the game-state.
- The phrase “decision model” generally describes any instructions, model, or technique for receiving queries and providing responses (in context). For one example, not intended to be limiting in any way, a decision model might include a technique for inference in a belief network.
- The phrase “behavior model” generally describes any information or other representation for describing behavior (in context) of an exemplar actor. For one example, not intended to be limiting in any way, a behavior model might include a belief network.
- The phrases “game designer” and “game software” generally describe, respectively, the entity providing the original game software and its concepts, and the original software capable of being compiled and executed as the game system or computer game.
- The phrases “behavior modification,” “dynamic learning,” “learned behavior” (such as for example, responses to stimuli within the game environment), “learning,” “learning by example,” “learning from a model,” “machine learning,” and “real-time learning” generally describe alteration of parameters of a behavior model, with the effect that the software agent controlled by that behavior model exhibits different behavior in the future compared with the past. In one embodiment, dynamic learning is performed by non-player characters in response to exemplary actions performed by player characters.
- The phrases “behavior strategies,” “cooperation,” “leadership,” “modeling behavior,” “training” and “retraining” generally describe types of exemplary behavior that might be presented, such as for example by a player character.
- The phrases “player,” or “user,” generally describe a person (or a team of persons) controlling a player character.
- The scope and spirit of the invention is not limited to any of these definitions, or to specific examples mentioned therein, but is intended to include the most general concepts embodied by these and other terms.
- System Elements
-
FIG. 1 shows a block diagram of a representation of a game system including context learning by software agents. - A
game 100 on agame system 105 includes agame engine 110, which in turn includesgame code 120, game-state 130, and one ormore software agents 140. - The
game code 120 includes a set of software processes for controlling the presentation of a computer game.Game code 120 typically is provided or coded by one or more game designers and programmers. - The game code preferably includes program structures such as data structures, data types, function entry points, function pointers, object data types, object methods, and the like. Some, all, none or different structure might be present depending on the way in which
game 100 is designed. - Although the
game code 120 is shown inFIG. 1 as a separate box, the other elements of thegame engine 110 typically are formed from portions ofgame code 120. - The game-
state 130 includes a set of state information, preferably used during operation of thegame engine 110. Although the game-state 130 is shown inFIG. 1 as a separate box, the game-state can encompass variables and data structures used by the other element of thegame engine 110. - The one or
more software agents 140 collectively include one ormore player characters 141 andnon-player characters 142. Preferably, aplayer character 141 includes a video object controlled by a human user 143 (or a team of such human users 143). For one example, not intended to be limiting in any way, “Lara Croft” (of the “Tomb Raider” Series) and “Jak” (of the “Jak and Daxter” game) are examples ofplayer characters 141. Preferably, anon-player character 142 includes a video object controlled by the system running the application software. For one example, not intended to be limiting in any way, the four roaming ghosts (“Blinky,” “Pinky,” “Inky,” and “Clyde”) in the classic “PacMan” game are examples ofnon-player characters 142. In some embodiments, it is possible to use game controllers or other input devices to affect the actions of at least some of thenon-player characters 142, such as for example the second “Ice Climbers” character in “Super Smash Brothers Melee” (for the Nintendo GameCube). - The program code for the
game engine 110 includes elements capable of specifying relevant information to other elements in thegame 100. - For a first example, not intended to be limiting in any way,
game engine 110 might include program structures having information relating toraw features 150. - The raw features 150 indicate important information about the
game 100, such as for example a location of a game objects, a point value of a game object, a measure of a wounded status of aplayer character 141 ornon-player character 142, and the like. In such cases, the program structures forraw features 150 might include C language code (or C++ language code, or code in another language) describing those data structures and how they can be referenced (such as for example a word offset within a data structure). - For a second example, not intended to be limiting in any way,
game engine 110 might include references to object methods capable of computingraw features 150, such as for example a distance between game objects, a direction of the most valuable object visible, and the like. In such cases, the program structures forraw features 150 might include C language code (or C++ language code, or code in another language) describing locations of those object methods or instructions for performing them, and how they can be referenced (such as for example a function-call address). - For example, not intended to be limiting in any way, a
raw feature 150 might include one or more of the following data: -
- A
raw feature 150 might include a data structure, for example from game-state 130, used by the game; that is, used by the simulation to determine what happens in the context of thegame 100. In one embodiment, the actual data structures used by the game are exposed to thelearner 165, so that those data structures might be directly read from and written into, without any special requirement for a data marshalling step (or similar step) or a buffer (or similar data structure) in which data is maintained after reading from that data structure or before writing into that data structure. For one example, not intended to be limiting in any way, there might be a data structure “playerInfo” including fields for that particular player's position, velocity, allegiance, power, health, whether that player is a player character or a non-player character, and the like. - A
raw feature 150 might include a list or an array of individual data structures, for example from game-state 130, used by the game; that is, used by the simulation to determine what happens in the context of thegame 100. For one such example, not intended to be limiting in any way, there might be an array of data structures “playerinfo [ ]” including fields similar to those described above, and for a plurality of players, whether player characters or non-player characters. The structure of theraw feature 150 as a list or an array has an advantage in that related information can be determined about the game-state 130 for one or more functions serving as inputs to derived features 160 (described below) or to one or more learners 165 (described below). For one such example, not intended to be limiting in any way, an array of such data structures might allow determination of information about player-to-player relationships, such as for example (for each player) which other player is the nearest enemy or friend, and the like. - A
raw feature 150 might include a list or an array of individual data structures, the individual data structures being similar to those described above, with the difference that the array relates to a set of objects other than players, such as an array of obstacles, an array of power-ups, and the like. For one such example, an array “nearest-BlueTarget [ ]” might include an identifier, location or pointer to another character (either player character or non-player character) who is the nearest blue target. - A
raw feature 150 might include a pointer to a descriptor or a function with the property of computing (or otherwise determining) a useful property of the game-state 130. For one example, not intended to be limiting in any way, araw feature 150 might include a function “isVisible (p1, p2)”, identified by a callback pointer or entry point, which provides a value describing whether player p2 is visible to player p1 (in response to game-state 130 such as for example whether player p2 has an invisibility cloak, player p1 is temporarily blinded, or can “see invisible,” in addition to considerations of line-of-sight, and the like). While it is possible in many cases for araw feature 150 of this type to be determined by thegame engine 110 as a derivedfeature 160, it might be advantageous for the game designer to provide that descriptor or function as araw feature 150, such as for example to take advantage of special knowledge of the game-state 130 or particular structure of the data structures used with that game-state 130.
- A
- As noted herein, there might be a very great many
raw features 150, such as for example a location of a game objects, a point value of a game object, a measure of a wounded status of aplayer character 141 ornon-player character 142, and the like. - The derived features 160 include a set of features derived in response to the
raw features 150, such as for example by calculating the values of those derivedfeatures 160 using the values of the raw features 150. A derivedfeature 160 might include a weighted sum of a set ofraw features 150, a function calculation in response to a set ofraw features 150, and the like. For one example, not intended to be limiting in any way, a derivedfeature 160 might include a “best path” from anon-player character 142 to a target object, with the distance and direction for that best path pre-computed for each point thereon. - A derived
feature 160 might be determined using one or more predefined calculations or operations performed on one or more raw features, raw features and derived features, or both raw features and derived 11 features. Some of the following examples are based on such predefined calculations or operations. - A derived
feature 160 might also or instead be determined using one or more learned or derived calculations or operations performed on one or more raw features, raw features and derived features, or both raw features and derived features. Some of the following examples are based on such predefined calculations or operations. - For example, not intended to be limiting in any way, a derived
feature 160 might include one or more of the following data: -
- A derived
feature 160 might include a Boolean value or a predicate operator, such as a determination whether some condition is true at the moment with regard to thegame 100. For one such example, not intended to be limiting in any way, there might be a predicate “p1->is Healthy” including a determination of whether a particular character p1 is considered healthy or not. In one embodiment, this particular derivedfeature 160 might be defined using the assignment “p1->is Healthy=IF (health>80% of maxPossibleHealth)”, where “health” and “maxPossibleHealth” areraw features 150. Similar derived features 160 might be constructed for concepts such as “isFarAway,” “isFriend,” and “is Enemy.” - A derived
feature 160 might include a Boolean value or a predicate operator defined in response to already-known predicates. For one such example, not intended to be limiting in any way, there might be a predicate “p1->is HealthyFriend” defined using the assignment “p1->is HealthyFriend=IF (p1->is Healthy AND p1->is Friend)”, where is Healthy and is Friend are themselves derived features 150. - A derived
feature 160 might include an iterator defined in response to an array of already-knownraw features 150 or derived features 160. For one such example, not intended to be limiting in any way, there might be an array playerinfo [ ]” as described above, including information relating to each player, and there might be an iterator “bluePlayerInfo [ ]” defined as the subset of all players in the array playerinfo [ ] also meeting the predicate operator is Blue. Similarly, using the iterator “bluePlayerInfo [ ],” another derivedfeature 160 such as “friendlyBluePlayerInfo [ ]” might be defined as the subset of all players in the array bluePlayerInfo [ ] also meeting the predicate operator isFriendly. Alternatively, the derivedfeature 160 such as “friendlyBluePlayerInfo [ ]” might be defined in response to onlyraw features 150 as the subset of the array playerinfo [ ] meeting the predicate operator (isBlue AND isFriendly). - A derived
feature 160 might include an element of an iterative operator by selecting one or more objects from that iterative operator. For one such example, not intended to be limiting in any way, there might be an iterative operator “bluePlayerInfo [ ]” as described above, with a selector nearestOne applied thereto to select the nearest blue player. Similarly, if the iterative operator were named “enemyPlayerInfo [ ]” and defined using a predicate operator is Enemy, there might be applied thereto the same selector nearestOne to select the nearest enemy player. - A derived
feature 160 might be defined in response to otherraw features 150, other derivedfeatures 160, or recursively defined in response to a variant form of the same derived feature 160 (such as for example a sorted version of an array or list). For one example, not intended to be limiting in any way, there might be a predicate operator “safeAtLast” defined in response to eitherraw features 150 or derivedfeatures 160, as for example using the assignment “safeAtLast=IF (nearestEnemy==Null)”. - A derived
feature 160 might be defined in response to alearner 165, with the effect that an output of a definedlearner 165 might be used as an input for defining a derivedfeature 160. For one example, not intended to be limiting in any way, alearner 165 might be defined to provide an array or list of the 10 choices recently made by a player character and the 10 choices predicted as most likely to be made by that player character for a “next move.” In such cases, a derivedfeature 160 might easily be defined in response to thatlearner 165, such as for example using the assignment “surpriseMove=IF (actualMove!=predictedMove)”.
- A derived
- One aspect of the invention is the provision of a selected set of derived features needed for the
learner 165 to be able to learn effectively. The inventors have found that it is non-obvious and unexpectedly useful to provide such a selected set of derived features. -
- First, this has the effect of allowing the game designer to not have to think about which derived features are going to be derived from the raw features (or other system state variables) of the game simulation, when designing or implementing that game simulation.
- Second, this has the effect of allowing the game designer to not have to design any code or data structures for providing those derived features, and also allowing the game designer to not have to design any code or data structures for computing those derived features.
- Third, this has the effect of allowing the game designer to not have to think about any learning aspects of the game simulation, and not having to design any code or data structures therefore.
- Fourth, this has the effect of allowing the game designer to call upon the
learner 165 provided by the invention to ask for actions to be taken by non player characters (or any other game element that might learn, such as for example an intelligent device).
- After reading this application, other and further advantages of providing a selected set of derived features will be clear to those skilled in the art. Moreover, these other and further advantages will be clear to be within the scope and spirit of the invention, and not to involve any undue experimentation or further invention.
-
Learner 165 is capable of providing dynamic learning for software agents ingame 100.Learner 165 is capable of learning from examples, capable of receiving requests from non-player characters and capable of responding to those requests with responses from which the non-player characters can readily derive one or more actions to take. In one embodiment, thelearner 165 gleans behavior from actions taken in context by a player character. This has the effect that when a non-player character queries that learner for a next action, the learner can provide a next action similar to one that might be taken by the player character (if the player character were directing the actions of that non-player character). - In one embodiment,
learner 165 makes predictions and generates responses to queries for given contexts based on learned behavior.Learner 165 can also be referred to as a simulation engine. - In
FIG. 1 ,learner 165 includes learning element(s) 170, behavior model(s) 180 and decision model(s) 190. The invention is not limited to learners constructed as shown inFIG. 1 . - Inputs to leaning element(s) 170 include
contexts 200 andtraining signal 210. Eachcontext 200 includes a set of values for the derived features 160 and possibly for the raw features 150. The number of possible contexts might be quite large. For example, not intended to be limiting in any way, if there are several hundredraw features 150, each of which might take on one of ten possible values, and there are several hundred derived features 160 determined in response to thoseraw features 150, the number ofpossible contexts 170 might be larger than 10 raised to the power of hundreds. -
Training signal 210 includes decisions made by one or more ofsoftware agents 140.Training signal 210 can also include comparisons between actions predicted bylearner 165 and actions actually taken by a software agent. Thus, learning element(s) 170 are trained based on what decisions one ormore software agents 140 make in various contexts. - In
FIG. 1 , learning element(s) 170 are the portions oflearner 165 that are primarily responsible for generating signals that result in machine learning. In particular, learning element(s) 170 generateupdates 175 for behavior model(s) 180. - A
behavior model 180 attempts to model behaviors exhibited bysoftware agents 140 invarious contexts 200. Behavior model(s) 180 preferably generate predictions based on context information. The predictions from behavior model(s) 180 attempt to approximate what a player character or trained non-player character would do in the given context. - In one embodiment, a
behavior model 180 includes a set of characteristics that (at least partially) predict the interaction of anon-player character 142 in various contexts during game-play. These characteristic could take the form of coefficients, weights, or other values - In the arrangement shown in
FIG. 1 , the context information for behavior model(s) 180 iscontexts 200 from derived features 160. Alternatively, the same or modified context information can be supplied to the behavior model(s) 180 from decision model(s) 190. For example, the decision model(s) could ask for a prediction about a context that might occur based on a current context but that is not actually occurring. - A
decision model 190 uses one or more behavior model(s) 180 to respond toqueries 220 fromgame engine 110, givencontexts 200. Accordingly, adecision model 190 is responsive to a set ofcontexts 200, and is capable of providing a decision for an action by anon-player character 142. For example, adecision model 190 might provide a decision for anon-player character 142 regarding whether, in a particular context, fight or flight is better for thatnon-player character 142. - The decision model(s) 190 can be viewed as wrapping the behavior model(s) 180, in the sense that a decision model takes a prediction from one or more behavior model(s) and can choose to adapt or modify the prediction to answer a query. Alternatively, the decision model(s) and behavior model(s) can be viewed as separate entities, in the sense that the behavior model(s) are more involved in updating and learning, whereas the decision model(s) are more involved in predicting behavior and responding to queries.
- The
learner 165 embodies one or more machine learning techniques. Accordingly, thelearner 165 would be ultimately responsive to theraw features 150 and derivedfeatures 160 in each ofcontexts 200, and thus responsive to eachcontext 200 as it occurs, to provide a response to a query from thegame engine 110. -
Multiple learning elements 170,behavior models 180 anddecision models 190 might be combined into one ormore learners 165, each for responding to queries in different situations. These multiple learners, learning elements, and/or models might control distinct aspects of the actions taken by a non-player character 142 (in a particular context 170). For one example, not intended to be limiting in any way, afirst learning element 180 might determine the fight-or-flight response (as described above) for thenon-player character 142, asecond learner 165 might determine what direction and how best to conduct flight when that response is chosen by thefirst learner 165, and athird learner 165 might determine how to conduct combat when that response is chosen by thefirst learner 165. - For example, not intended to be limiting in any way, a
learner 165 might be responsive to include one or more of the following queries and provide one more of the following responses: -
- A query could be “what action should be taken next given a past, current, or predicted context.” The action could be a direction vector to travel, a speed to travel, a discrete action choice (e.g. “punch”, “shoot”, “jump”, a force to apply, or any other action you can imagine a player character or NPC having to choose in a game. The action could also be a “high-level” action that is translated down into game-level action by some other component. For example, not intended to be limiting in any way, the learner could respond with a goal location, a path planner then plans a path to the goal, and a path follower generates the low-level game actions to follow the path.
- A query could be “what event is likely or possible to happen next” (such as for example, predicting a selected next event or a selected action by another character). The response could be an action that a player character or non-player character is likely take in a given context, a predicted change in context, or the like.
- A query could be “what is a ranking for a set of possibilities.” These possibilities could include possible player character or non-player character actions, possible current or future contexts, or the like. The rankings could be based on importance, likelihood of success, amount of time required, or the like, relative to some player character or non-player character goal or some game context.
- A query could be for a sample from a distribution over a set of actual or possible results. In one embodiment, it is not necessary for the query maker to know the actual probability distribution to obtain a response.
- A query could be “what is the probability, or conditional probability of a given event.” In one embodiment, it is not necessary for the query maker to know the actual probability distribution to obtain a response.
- A query could be “what is the probability distribution (in whole or in part) for a set of events.” In one embodiment, it is not necessary for the query maker to know the actual probability distribution for each individual event to obtain a response.
- A query could be “what is the location of similar objects within some set” given criteria by which the grouping is made.
- A query could be “what are the top N most important factors in making a given decision,” possibly with respect to a given phenomena, for example that explain the differences between 2 different answers to some other query.
- A query could be “how certain or uncertain is the model about the query result being returned. In one embodiment, it is not necessary for the query maker to know how the query result is actually determined to obtain a response.
- A query could be “what is the most critical information (input samples) that can be collected to reduce the uncertainty of a given query or class of queries.” In one embodiment, it is not necessary for the query maker to know how the query result is actually determined to obtain a response.
- A query could be “what is the most/least common result over a distribution of results.” In one embodiment, it is not necessary for the query maker to know the actual probability distribution for each individual result, or how the query result is actually computed, to obtain a response.
- A query could be “why was a given action predicted.” In one embodiment, it is not necessary for the query maker to know the actual probability distribution for each individual predicted action.
- A query could be “what actions should be taken” to achieve a given goal in a given context.
- A query could be “what actions would one player or non-player character be likely to perform in place of another player or non-player character,” as well as a listing or ranking of probabilities for those actions.
- Other queries could include (from the perspective of a character) “what would I do if I were you,” “what might I do in this other situation,” “what are my top N choices,” and the like.
- As
learner 165 is providingresponses 230, thegame engine 110 compares those responses with the actions of a target whose behavior thenon-player character 142 seeks to imitate. For one example, not intended to be limiting in any way, the target might include aparticular player character 141, with the effect that the decisions made by the learner 165 (as requested by the game engine 110) are compared with actual decisions made by theplayer character 141. - In the foregoing description of
FIG. 1 ,learner 165 learns by having learning element(s) 170 update behavior model(s) 180, which in turn affect decision model(s) 190. The invention is not limited to this particular arrangement. Other designs of elements and modules that result in machine learning or other forms of artificial intelligence can be used with the invention. In those other designs, each of learning element(s) 170, behavior model(s) 180, and decision model(s) 190 might or might not be present, might have different responsibilities, or might be replaced with one or more other elements or modules. - Learning
- A
human player 143 generally makes better decisions the next time they encounter a situation based on the success of prior decisions and their outcomes. Simple Artificial Intelligence in computer games often does not learn because it generally retains no prior knowledge of past experiences, and when it does, it does not learn from that experience so as to act differently in that context a next time. - Enabling a game that learns to interact more like human beings keeps the game fresh for the
human user 143. For example, “PacMan” was a very popular game in the early eighties. After publication of known “play patterns” for itsnon-player characters 142, anyone who knew the patterns could potentially score an unlimited number of points. Essentially, the game was beaten and gamers looked to other games that presented a challenge. - The
learner 165 enables techniques whereby a game designer or programmer unskilled in artificial intelligence can take advantage of more powerful learning techniques, overcoming these short-falls of prior games that do not learn or that learn through simple AI techniques implemented by game designers and programmers. - In one embodiment, decisions made by the learner 165 (as requested by the game engine 110) are compared with actual decisions made by the
player character 141.Training signal 210 can represent the actual decisions, a comparison ofresponses 230 to those decisions, or some other measure of actual decisions.Learning element 170 oflearner 165 can use this training signal to updatebehavior model 180. - After reading this application, those of ordinary skill in the art would recognize that
learner 165 can implement a predictive technique in whichlearner 165 attempts to predict thetraining signal 210 for eachcontext 200. When thelearner 165 is sufficiently good at predicting that training signal, it might be said that thelearner 165 “learned to imitate” atarget player character 141. - In one embodiment, the behavior model(s) 180 of
learner 165 might use weighted averages, median calculations, or other statistical measures to make predictions. Learning element(s) 170 could update the coefficients or other values used to implement the statistical measures based oncontexts 200 andtraining signal 210. - Preferably, the weights or coefficients for a
behavior model 180 are real-valued coefficients or other values as precise as available using machine arithmetic. However, in the context of the invention, there is no particular requirement for selecting the coefficients or statistical measure from any particular number field. - The inventors have found that an exponential gradient technique for machine learning provides a reasonably quick, sufficient method by which the coefficients of learning
element 170 can be updated. - Other techniques can be used by
learner 165 instead of or in addition to the exponential gradient technique. Examples of such other techniques include but are not limited to a Bayesian classifier, a neural network, an expert system, a hidden Markov model, a partially observable Markov decision process model, etc. - After reading this application, those of ordinary skill in the art would recognize that many variations on the
learner 165 are within the scope and spirit of the invention, and that those variations would not require undue experimentation or new invention. Some examples, not intended to be limiting in any way, include: -
- training an individual
non-player character 142 in response to an individualtarget player character 141, such as for example to imitate thatplayer character 141, to do the opposite of thatplayer character 141, to act favorably toward thatplayer character 141, to act inimically toward thatplayer character 141, to act neither like nor dislike (that is, “orthogonal to”) thatplayer character 141, or in general any transformation of the above, such as for example to act like thatplayer character 141 but more slowly; - training a group of
non-player characters 142 in response to an individualtarget player character 141, such as for example to (imitate, do the opposite of, favor, disfavor, act orthogonally to, or some transform of the above) with regard to decisions made by thatplayer character 141—this would include training a group of “followers” for thatplayer character 141; - training one or more
non-player characters 142 in response to a group oftarget player characters 141, where each decision or action made by anindividual player character 141 might be considered in the training process, with the effect that one or more of thenon-player characters 142 act in response to a statistical measure of the actions taken by those player characters 141 (for one such example, the training signal might be considered to be modal action taken by the set ofindividual player character 141 in that context, that is, whether most of them run away or stay to fight); - training one or more
non-player characters 142 in response to a set of results (short-term, long-term, or otherwise) produced by a target, where that target might be one ormore player characters 141, one or more non-player characters 142 (possibly including the ones being trained), or some combination thereof—this would include a technique for unsupervised learning by one or morenon-player characters 142 in thegame 100; - training one or more
non-player characters 142 in response to an aggregate, average, or some other statistical sampling of actions taken by a group of player characters or non-player characters, for example to train the non-player characters based on a collective strategy used by a group of player or non-player characters (for one such example, the training signal for a first set of non-player characters might include the actions of a player character for a first time duration, while the training signal for a second set of non-player characters might include the actions of a player character for a second time duration, with the effect that the second set of non-player characters acts differently from the first set of non-player characters if the player character acted differently during the second time duration than during the first time duration); - training one or more subsets of
non-player characters 142 at different times, with the effect that one or more non-player characters in a subset can later be designated to perform a certain action, to be used in a selected context, or the like, either by the game system or by a player; - retraining or repartitioning one or more subsets of non-player characters that were trained at different times.
- training an individual
- In some embodiments, the
game engine 110 might provide ahuman user 143 with access to attributes within anon-player character 142 generated by thelearner 165, such as for example to correct any flaws, or such as for example to add flaws, such as when thenon-player character 142 plays the game too well and is hard to beat. - In various embodiments,
non-player characters 142 might start with any one of the following: -
- a blank
slate behavior model 180—that is, no important information about thegame 100; - a
basic behavior model 180—that is, some basic information about thegame 100, including possibly some rules to prevent thenon-player character 142 from doing anything exceedingly stupid; - an
advanced behavior model 180—that is, significant information about thegame 100, possibly ranked by “experience” or “level” or some other measure of knowledge about thegame 100.
- a blank
- In various embodiments,
non-player characters 142 might be coupled to a training signal for learning, such as any one of the following: -
- no training signal
- a
player character 141 as the training signal, such as for example to learn to imitate the knowledge and style of thatplayer character 141; - reinforcement from the
game 100 as the target, such as for example a form of unsupervised learning for thenon-player character 142, as described above; - reinforcement from another
non-player character 142 as the training signal, such as for example a set of “experienced”non-player characters 142 as described below—in such embodiments,non-player characters 142 might alternatively use each other as target training signals in a simulated contest among or between thosenon-player characters 142, providing a second form of unsupervised learning for thenon-player characters 142.
- In various embodiments,
non-player characters 142 might be coupled to alearner 165 for learning, such as any one of the following: -
- no learning
- learning for an initial duration, after which the
non-player character 142 has acquired some of the knowledge and style of the target, but not so much as to look like a copy thereof—the inventors have found that even a few seconds of regular play can bring anon-player character 142 to a point of looking very much like itstarget player character 141; - learning for a substantial duration, after which the
non-player character 142 has acquired a substantial amount of the knowledge and style of the target, possibly enough to directly challenge the knowledge and style of thatplayer character 141; - continual or periodic learning, in which the
non-player character 142 continues to acquire more of the knowledge and style of the target.
- A production run of a
game 100 or application within thegame 100 might include at least somenon-player characters 142 for each degree of development in theirbehavior model 180. - In one embodiment, behavior models for the
non-player character 142 might be saved for later use, either in thesame game 100 or in another copy of that game 100 (that is, behavior models might be communicated or carried on physical media from onegame 100 to another game 100). However, in the context of the invention, there is no particular requirement for actually saving any behavior models for any or all of thenon-player characters 142. - Storage
- The
storage element 240 includes a device for storing electronic data including but not limited to a hard disk drive, an optical drive, a silicon chip storage element (smart media card, video game external storage card, random access memory). Code and data for thegame 100 can be saved to thestorage 240. - In one embodiment, data that is generated by the
learner 165 might be saved and loaded, either on demand, or in response to particular inputs (such as for example a game controller button). This data might be used as aninitial behavior model 180, that is, pre-training, or might be used as astatic behavior model 180, that is, post-training. Thebehavior model 180 might be incorporated into one ormore software agents 140, such as for examplenon-player characters 142. In some embodiments, new non-player characters might be generated with theirinitial behavior model 180 at least partially specified by thehuman player 143. - Applications
- The invention enables a multitude of new modalities, mechanisms and options (collectively referred to as techniques below) for game designers. Examples, not intended to be limiting in any way, are described below. Application of the invention to these modalities, mechanisms and options would not require further invention or undue experimentation. These applications include techniques that support learning behavior either during actual game play, in a batch mode, or a combination thereof.
-
- A technique that supports a “train by playing” modality whereby a player character's activities while playing the game is used to train non-player-characters. Entities such as followers (i.e., non-player characters) can learn not only the actions the player tends to take, but the contexts in which to apply them. Learning can continue over the course of the game.
- A technique that supports creating teams wherein individual members of the team are trained individually by a player using the “train by playing” modality.
- A technique that supports in-play controlling and/or influencing actions of other non-player-entities in a game. Entities include but are not limited to non-player-characters, and the active control elements of the virtual game world. Entities can be followers, friends or enemies.
- A technique that supports controlling and/or influencing the actions of a group of followers that learn both the actions the player takes, as well as the context in which the player takes those actions in 2D and 3D tactical/strategic action and fighting games.
- A technique that supports controlling and/or influencing entities that can recognize millions of different contexts and behave appropriately according to how they have observed the player behaving.
- A technique that supports controlling and/or influencing entities that can adapt and evolve their behavior on-the-fly and over time as the player's behavior changes.
- A technique that supports controlling and/or influencing entities that can learn tactical or high-level strategic play from the player.
- A technique that supports controlling and/or influencing the actions of teammates in sports games. The sport maybe a depiction of a known sport, or a sport with rules concocted for the video game.
- A technique that supports controlling and/or influencing entities that learn quickly and for whom learning can continue indefinitely.
- A technique that supports controlling and/or influencing entities that can learn individual skills and/or how to co-operate in large and small groups.
- A technique that supports controlling and/or influencing entities either during a single critical decision, or all the time, or anything in-between.
- A technique that supports creating behavior for entities that can be saved, restored, replayed and/or traded.
- A technique that supports personalizing a player's experience of the game.
- A technique that supports controlling and/or influencing entities that can learn to make tactical decisions like, such as but not limited to: timing, spacing, approach, positioning and selection of opening moves and follow-up moves for attacks, blocks, and backing off; target selection, including double-teaming; direction and style of movement, where and when to dodge and flee; selection of a level of aggression; ordering and timing of decisions; deciding to flee or fight.
- A technique that supports controlling and/or influencing target selection decisions, such as but not limited to: aiming for the heart of a cluster; aiming at stragglers; ignoring mobs and go for a leader or vice-versa; selecting which power-ups to go after, which crates to open, with what priority, and the like; selecting which rooms to in, which doors to try, and the like.
- A technique that supports controlling and/or influencing entities that initially have either no knowledge, very little knowledge, lots of knowledge or anything in-between and learn from the player.
- A technique that supports controlling and/or influencing a single companion, a core group of followers, whole armies, or anything in between.
- A technique that supports controlling and/or influencing entities in single player mode, or in head-to-head events that show off training and leadership skills.
- A technique that supports controlling and/or influencing entities whose errors can be corrected by demonstration.
- A technique that supports controlling and/or influencing entities in different groups in different ways. That is, some user interface element, or pre-defined condition, is used to specify to which group the training signal is applied and when.
- A technique that supports controlling and/or influencing entities whose behavior can evolve over multiple scenarios, and multiple games.
- A technique that supports controlling and/or influencing entities at the formation level. For example, not intended to be limiting in any way, a group of entities can learn: What is the shape of a given squad—a line, a wedge, a square? When does a given squad choose to engage enemies, and when does it flee? Does every soldier attack for himself, or should he hold formation and coordinate attacks?
- A technique that supports optionally controlling and/or influencing entities that can learn from the player with commands like: “Attention!,” “Dismissed!,” “Scatter!,” or “Form up!”
- A technique that supports creating non-player-entities that can explain their behavior.
- A technique that supports creating non-player-entities that can negotiate.
- A technique that supports sharing trained avatars (i.e., characters), for example to permit later selection of a best-trained avatar for each head-to-head encounter.
- A technique that supports allowing a non-player character to emulate behavior of expert or even world-champion players.
- A technique that supports training several generals at once on the battlefield, either by example or by swapping in to control them.
- A technique that supports allowing a player to create avatars as opponents for other players to compete with.
- A technique that supports creating and evolving a knowledge base per player, for example to be loaded up for competitive reasons.
- Moreover, in the context of the invention, there is no particular requirement that the player themself cannot learn from non-player characters. For example the invention enables a technique that supports training a new player character by providing hints or indicators of what other player characters (or scripts designed for non-player characters would do in a similar context. Application of the invention to these techniques would not require further invention or undue experimentation.
- Dead Reckoning
- Another technique enabled by the invention includes “deal reckoning,” in which a game (or other virtual simulation) might be performed on more than one computing device. For one example, not intended to be limiting in any way, a game might be played using two logically or physically remote computing devices A (including controlling a first player character X at computing device A) and B (including controlling a second player character Y at computing device B). Each computing device might use a prediction function predictLocation (character, futureTime) to predict a location of that selected character at that selected futureTime.
- If each computing device has access to a substantially identical function predictLocation ( ), the computing devices might exchange only differences between actual values and predicted values. For one example, not intended to be limiting in any way, if computing device A predicts character X's future location at a time t1 as XA=predictLocation (X, t1), while computing device B predicts character X's future location at the time t1 as XB=predictLocation (X, t1), the two computing devices each know approximately where character X should be, that is, XA≈XB. If computing device A knows that character X is in a substantially different location, that is, computing device A's computation of XA−actualLocation (X, t1)>some minimum threshold tolerance, computing device A knows it needs to send information to computing device B to update the latter's value of actualLocation (X, t1).
- In one embodiment, the predictor predictLocation ( ) can be improved by learning in context, either at computing device A, computing device B, or both. In such embodiments, computing devices A and B might need (but alternatively might dispense with) sending information to update each others' decision model for predictLocation ( ), but the inventors expect that total communication traffic between by computing devices A and B would be substantially reduced.
- After reading this application, those skilled in the art would recognize that predictLocation ( ) has a clear training signal, as whenever it is wrong by more than some minimum threshold tolerance, it is corrected to provide the actual location value.
-
FIG. 2 shows a block diagram of an example of context learning by software agents. The invention is not limited in any way to the details of this example. - Raw features 150 are shown in
FIG. 2 as including various values and indications forgame code 120. A learner according to the invention generates the derived features 160 from theraw features 150 and possibly from other derived features 160. - In the example shown in
FIG. 2 , the game code is for a simulator such as a flight simulator. The variousraw features 150 relate to information about the flight simulation. The particular raw features shown inFIG. 2 are meant to be illustrative only; actual code for a flight simulator might use all, some, none, or different raw features. - The derived features 160 in
FIG. 2 represent relationships between some ofraw features 150. Derived features 160 according to the invention are not limited to such relationships. Furthermore, the particular relationships shown inFIG. 2 are meant to be illustrative only; actual derived features for a flight simulator might use all, some, none, or different derived features. - In order to generate the derived features, the learner must know where in the
game code 120 to find those features. This information preferably is extracted from the code, preferably based on information supplied by the game's developer or designer, and based on templates supplied for this purpose. - The derived features 160 form one or
more contexts 200. These contexts are supplied to learning element(s) 170 and behavior element(s) 180, where they are used to generatepredictions 185. - The contexts also are supplied to decision model(s) 1909, where they are used in conjunction with
predictions 185 to respond toqueries 220 withresponses 230. These queries and responses are used bygame code 120 to incorporate the learning capabilities of the invention into the code. - The game code supplies training signals 210 to learning element(s) 170, which use the signals to update behavior element(s) 180. The behavior element(s) in turn affect the operation of decision model(s) 190.
- Method of Operation
-
FIG. 3 shows a process flow diagram of a method of operating in a system for context learning by software agents. - A
method 300 is performed by thegame 100. Although themethod 300 is described serially, the flow points and steps of themethod 300 can be performed by separate elements in conjunction or in parallel, whether asynchronously or synchronously, in a pipelined manner, or otherwise. There is no particular requirement that themethod 300 must be performed in the same order in which this description lists flow points or steps, except where explicitly so indicated. - Associating, Updating and Using Behavior Models
- At a
flow point 301, themethod 300 is ready to begin associating and updating behavior model(s) 180 withsoftware agents 140. - At a
step 302, thegame 100 is activated, and themethod 300 begins to operate in conjunction or combination with thegame 100. - At a
step 303, themethod 300 determines if there should be anynon-player characters 142 with pre-loaded behavioral and decision model(s). If so, themethod 300 performs the next step (the step 304), and continues with the step thereafter (the step 305). If not, themethod 300 skips the next step (the step 304), and continues with the step thereafter (the step 305). - At a
step 304, themethod 300 receives one or morebehavioral models 180 anddecision models 190 from thestorage element 240, and incorporates or associates those models tonon-player characters 142 as determined in the previous step (the step 303). - At a
step 305, themethod 300 determines if any non-player characters 142 (or other software agents 140) should be coupled to alearner 165 for learning. If so, the method performs the next step (the step 306), and continues with the step thereafter (the step 307). If not, themethod 300 skips the next step (the step 306), and continues with the step thereafter (the step 307). - At a
step 306, themethod 300 couples one or morenon-player characters 142 tolearners 165, as determined in the previous step (the step 305). - At a
step 307, themethod 300 updates thebehavior models 180, for example usinglearning elements 170 acting in response tocontexts 200, which in turn are based onraw features 150 and derivedfeatures 160, and in response to training signal(s) 210. - In one embodiment, the contexts might include a set of changes from earlier recent contexts. However, in the context of the invention, there is no particular requirement for any special format for this information. Either all of
contexts 200, or parts thereof, or a set of differences, or some of them, might be used. - In one embodiment, the training signals could be actions actually taken by a player or non-player character, a comparison of actions taken with earlier responses from leaner(s) 165 or predictions by behavior model(s) 180, or some other type of feedback.
Learning elements 170 update behavior model(s) 180 responsive to these training signals, for example by updating coefficients, weights, or other values used by the behavior model(s). - At a
step 308, themethod 300 receives queries, for example for what action a non-player character should take, from one or more of thesoftware agents 140. - At a
step 309, the behavior model(s) 180 make predictions about what action a player character or some other character would take in the current context. - At a
step 310, the decision model(s) 190 make decisions responsive to the predictions. - At a
step 311, a response to the query is sent to the querying software agent responsive to the decisions. - In a training mode, less than all of these steps might be performed. For example, steps 310 and 311 might be skipped in a training mode. Conversely, if the learner(s) are considered to be fully trained,
step 307 might be skipped. Other variations are possible. - Those of
steps 307 to 311 that are performed preferably are repeatedly executed so as to facilitate learning by the software agents. Asraw features 150 and derivedfeatures 160 change,contexts 200 change, permitting the agents to learn behaviors for different contexts. This continues until the game ends insteps - Interrelation of Method Parts
- After reading this application, those skilled in the art will recognize that many variations are possible that would be workable, that such variations are within the scope and spirit of the invention, and that such variations would not require undue experimentation or further invention.
- For example, not intended to be limiting in any way, the following aspects of the
method 300 would be clear to those skilled in the art: -
- Each of
steps 307 to 311 can be, and preferably are, performed in parallel. This has the effect that themethod 300 can update thelearners 165 concurrently with using thoselearners 165 to make decisions on behalf ofnon-player characters 142. - In the context of the invention, there is no particular requirement for learning to start and stop when the game starts or stops. For example, learning might be conducted for only a short duration after the game starts, or at pre-selected time durations during the game, or at periodic time durations, or at time durations responsive to the game itself (such as for example, performing learning for combat skills only during intense combat).
- In the context of the invention, it is possible for learning to be conducted in context at one or more levels of generality. For one example, not intended to be limiting in any way,
non-player characters 142 can learn to imitate player characters' 142 skill at identifying the most dangerous enemy (e.g., not always the one that is hardest to defeat or does the most damage). Another example, also not intended to be limiting in any way,non-player characters 142 can learn to imitate player characters' 142 skill at determining when to initiate and when to break off combat (e.g., when health is too low for the remaining enemies, or when the remaining enemies appear too powerful to defeat without substantial risk). Another example, also not intended to be limiting in any way,non-player characters 142 can learn to imitate player characters' 142 degree of daring or risk-taking (e.g., whether particular risks are “too risky” or not). Another example, also not intended to be limiting in any way,non-player characters 142 can learn to imitate player characters' 142 goal preferences (e.g., whether to improve health or to find treasure to win the game, whether to prepare defensively against damage or offensively to defeat enemies, and the like). Another example, also not intended to be limiting in any way,non-player characters 142 can learn to imitate player character's 142 approach to goal-making or degree of intentionality of play (such as for example, whether to go out “looking for trouble,” or to focus planning and effort on a specific enemy or prize). After reading this application, hose skilled in the art would recognize that all of these concepts are within the scope and spirit of the invention, would be workable, and would not require undue experimentation or further invention. - The more the game is played, the more experiential learning
non-player characters 142 gain. This enhances overall game play and rewards thehuman user 143 for time invested in playing the game. - Most of the currently popular game consoles promote collaborative or competitive play. For example, the Sony PlayStation 2 console allows players to connect over a network to play as a team against other teams or to play against each other as enemies. Using the invention, a host of scenarios become possible. For example, but without limitation, a single human user can be a team that includes a
player character 141 and one or morenon-player characters 142. - In addition to collaborative or competitive games, there are also games representative of group activities, such as sports teams, corporate activity, or military units. For one example, not intended to be limiting in any way, a first possible use of the invention is for rapidly providing teams of
non-player characters 142 with particular characteristics, such as for example a platoon of extremely loyal soldiers. For another example, again not intended to be limiting in any way, a second possible use of the invention is for providing teams ofnon-player characters 142 who imitate theplayer character 141 and thus allow theplayer character 141 to demonstrate (or learn) leadership. For another example, again not intended to be limiting in any way, a third possible use of the invention is for providingnon-player characters 142 who rapidly learn the stratagems of theplayer character 141 and rapidly learn to neutralize simple tricks or quirks of thegame 100.
- Each of
- Although preferred embodiments are disclosed herein, many variations are possible which remain within the concept, scope, and spirit of the invention. These variations would become clear to those skilled in the art after perusal of this application.
- As noted above, the invention enables a multitude of new modalities, mechanisms and options (collectively referred to as techniques below) for game designers. Examples, not intended to be limiting in any way, are described below. Application of the invention to these modalities, mechanisms and options would not require further invention or undue experimentation.
-
- For example, not intended to be limiting in any way, the invention supports techniques by which multiple non-player characters can learn from multiple player characters, for example by (a) designating one of the player characters as a target signal to be imitated by one or more non-player characters, (b) designating a statistical value in response to a set of player characters as a target signal to be imitated by one or more non-player characters.
- For example, not intended to be limiting in any way, the invention supports providing generalized artificial intelligence or machine learning techniques that are responsive to an exemplar, such as for example by generating a model of desired behavior without explicitly programming that behavior.
- After reading this application, those skilled in the art would recognize that the techniques described herein provide an enabling technology, with the effect that heretofore advantageous features can be provided that heretofore were substantially infeasible.
- After reading this application, those skilled in the art will recognize that these alternative embodiments and variations are illustrative and are intended to be in no way limiting.
Claims (42)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/885,495 US7296007B1 (en) | 2004-07-06 | 2004-07-06 | Real time context learning by software agents |
PCT/US2005/024139 WO2006014560A2 (en) | 2004-07-06 | 2005-07-06 | Real time context learning by software agents |
US11/929,170 US7636701B2 (en) | 2004-07-06 | 2007-10-30 | Query controlled behavior models as components of intelligent agents |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/885,495 US7296007B1 (en) | 2004-07-06 | 2004-07-06 | Real time context learning by software agents |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/929,170 Continuation US7636701B2 (en) | 2004-07-06 | 2007-10-30 | Query controlled behavior models as components of intelligent agents |
Publications (2)
Publication Number | Publication Date |
---|---|
US20070260567A1 true US20070260567A1 (en) | 2007-11-08 |
US7296007B1 US7296007B1 (en) | 2007-11-13 |
Family
ID=35787655
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/885,495 Expired - Fee Related US7296007B1 (en) | 2004-07-06 | 2004-07-06 | Real time context learning by software agents |
US11/929,170 Expired - Fee Related US7636701B2 (en) | 2004-07-06 | 2007-10-30 | Query controlled behavior models as components of intelligent agents |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/929,170 Expired - Fee Related US7636701B2 (en) | 2004-07-06 | 2007-10-30 | Query controlled behavior models as components of intelligent agents |
Country Status (2)
Country | Link |
---|---|
US (2) | US7296007B1 (en) |
WO (1) | WO2006014560A2 (en) |
Cited By (50)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060200588A1 (en) * | 2005-02-04 | 2006-09-07 | Samsung Electronics Co., Ltd. | Method and apparatus for determining load of input/output command in input/output subsystem |
US20070136745A1 (en) * | 2005-12-08 | 2007-06-14 | International Business Machines Corporation | Brokering of personalized rulesets for use in digital media character replacement |
US20070132780A1 (en) * | 2005-12-08 | 2007-06-14 | International Business Machines Corporation | Control of digital media character replacement using personalized rulesets |
US20080097948A1 (en) * | 2004-07-06 | 2008-04-24 | Ailive, Inc. | Real Time Context Learning by Software Agents |
US20080115086A1 (en) * | 2006-11-15 | 2008-05-15 | Yahoo! Inc. | System and method for recognizing and storing information and associated context |
US20080114758A1 (en) * | 2006-11-15 | 2008-05-15 | Yahoo! Inc. | System and method for information retrieval using context information |
US20080115149A1 (en) * | 2006-11-15 | 2008-05-15 | Yahoo! Inc. | System and method for providing context information |
US20080154912A1 (en) * | 2006-12-22 | 2008-06-26 | Yahoo! Inc. | Method and system for locating events in-context |
US20090307671A1 (en) * | 2008-06-06 | 2009-12-10 | Cornell University | System and method for scaling simulations and games |
US7636645B1 (en) | 2007-06-18 | 2009-12-22 | Ailive Inc. | Self-contained inertial navigation system for interactive control using movable controllers |
US20100029384A1 (en) * | 2008-07-22 | 2010-02-04 | Sony Online Entertainment Llc | System and method for physics interactions in a simulation |
US20100121808A1 (en) * | 2008-11-11 | 2010-05-13 | Kuhn Michael J | Virtual game dealer based on artificial intelligence |
US20110250575A1 (en) * | 2010-04-13 | 2011-10-13 | enVie Interactive LLC | System And Method For Providing A Visual Representation Of A User Personality Within A Virtual Environment |
US8821290B2 (en) | 2011-01-12 | 2014-09-02 | Kabushiki Kaisha Square Enix | Automatic movement of disconnected character in network game |
US20150051000A1 (en) * | 2013-08-16 | 2015-02-19 | Disney Enterprises, Inc. | Perception system |
US9033796B2 (en) | 2011-01-12 | 2015-05-19 | Kabushiki Kaisha Square Enix | Automatic movement of player character in network game |
US20160114247A1 (en) * | 2014-10-23 | 2016-04-28 | Here Global B.V. | Method and apparatus for providing privacy policy generation based on in-game behavior data |
US10042514B2 (en) * | 2014-10-30 | 2018-08-07 | Microsoft Technology Licensing, Llc | Typeahead features |
US20180314963A1 (en) * | 2017-04-19 | 2018-11-01 | AIBrain Corporation | Domain-independent and scalable automated planning system using deep neural networks |
US10286322B1 (en) | 2016-01-25 | 2019-05-14 | Electronic Arts Inc. | System and method for determining and executing actions in an online game |
US20190156565A1 (en) * | 2016-04-28 | 2019-05-23 | Verizon Patent And Licensing Inc. | Methods and Systems for Distinguishing Objects in a Natural Setting to Create an Individually-Manipulable Volumetric Model of an Object |
US10300379B1 (en) | 2015-12-28 | 2019-05-28 | Electronic Arts Inc. | Facilitating contextual game notifications and smart action options |
US10394414B1 (en) * | 2013-07-19 | 2019-08-27 | Kabam, Inc. | Facilitating automatic execution of user interactions in a virtual space |
CN110339569A (en) * | 2019-07-08 | 2019-10-18 | 深圳市腾讯网域计算机网络有限公司 | Control the method and device of virtual role in scene of game |
GB2574650A (en) * | 2018-06-14 | 2019-12-18 | Sony Interactive Entertainment Inc | System and method for generating an input signal |
US10512846B2 (en) * | 2017-03-07 | 2019-12-24 | Sony Interactive Entertainment LLC | Emulating player behavior after player departure |
US20200197811A1 (en) * | 2018-12-18 | 2020-06-25 | Activision Publishing, Inc. | Systems and Methods for Generating Improved Non-Player Characters |
US10812510B2 (en) * | 2018-01-12 | 2020-10-20 | The Boeing Company | Anticipatory cyber defense |
US10881964B1 (en) * | 2018-09-13 | 2021-01-05 | Electronic Arts Inc. | Automated detection of emergent behaviors in interactive agents of an interactive environment |
CN112218693A (en) * | 2018-03-23 | 2021-01-12 | 索尼互动娱乐有限责任公司 | Voice help system using artificial intelligence |
US20210027164A1 (en) * | 2018-01-22 | 2021-01-28 | Apple Inc. | Objective-effectuators in synthesized reality settings |
US10933322B2 (en) * | 2015-06-30 | 2021-03-02 | FanDuel Limited | Method and system for encouraging a person to affirmatively engage in an activity |
US10946280B2 (en) | 2017-03-08 | 2021-03-16 | Sony Interactive Entertainment LLC | In-game reactions to interruptions |
US10946281B2 (en) * | 2019-03-29 | 2021-03-16 | Nvidia Corporation | Using playstyle patterns to generate virtual representations of game players |
US11110353B2 (en) * | 2019-07-10 | 2021-09-07 | Electronic Arts Inc. | Distributed training for machine learning of AI controlled virtual entities on video game clients |
CN113648660A (en) * | 2021-08-16 | 2021-11-16 | 网易(杭州)网络有限公司 | Behavior sequence generation method and device for non-player character |
US11276216B2 (en) | 2019-03-27 | 2022-03-15 | Electronic Arts Inc. | Virtual animal character generation from image or video data |
US11351459B2 (en) | 2020-08-18 | 2022-06-07 | Activision Publishing, Inc. | Multiplayer video games with virtual characters having dynamically generated attribute profiles unconstrained by predefined discrete values |
US11369880B2 (en) | 2016-03-08 | 2022-06-28 | Electronic Arts Inc. | Dynamic difficulty adjustment |
US11406899B2 (en) | 2019-03-27 | 2022-08-09 | Electronic Arts Inc. | Virtual character generation from image or video data |
US11413539B2 (en) | 2017-02-28 | 2022-08-16 | Electronic Arts Inc. | Realtime dynamic modification and optimization of gameplay parameters within a video game application |
US11413536B2 (en) | 2017-12-22 | 2022-08-16 | Activision Publishing, Inc. | Systems and methods for managing virtual items across multiple video game environments |
US11458399B2 (en) | 2016-12-30 | 2022-10-04 | Electronic Arts Inc. | Systems and methods for automatically measuring a video game difficulty |
US11524234B2 (en) | 2020-08-18 | 2022-12-13 | Activision Publishing, Inc. | Multiplayer video games with virtual characters having dynamically modified fields of view |
US11524237B2 (en) | 2015-05-14 | 2022-12-13 | Activision Publishing, Inc. | Systems and methods for distributing the generation of nonplayer characters across networked end user devices for use in simulated NPC gameplay sessions |
US11532172B2 (en) | 2018-06-13 | 2022-12-20 | Electronic Arts Inc. | Enhanced training of machine learning systems based on automatically generated realistic gameplay information |
US20230034222A1 (en) * | 2021-07-28 | 2023-02-02 | Blizzard Entertainment, Inc. | Initial results of a reinforcement learning model using a heuristic |
WO2023039564A1 (en) * | 2021-09-12 | 2023-03-16 | Sony Interactive Entertainment Inc. | Using data from a game metadata system to create actionable in-game decisions |
US11712627B2 (en) | 2019-11-08 | 2023-08-01 | Activision Publishing, Inc. | System and method for providing conditional access to virtual gaming items |
US11763143B2 (en) | 2017-04-19 | 2023-09-19 | AIBrain Corporation | Adding deep learning based AI control |
Families Citing this family (92)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8678902B2 (en) | 2005-09-07 | 2014-03-25 | Bally Gaming, Inc. | System gaming |
US7161579B2 (en) | 2002-07-18 | 2007-01-09 | Sony Computer Entertainment Inc. | Hand-held computer interactive device |
US8797260B2 (en) | 2002-07-27 | 2014-08-05 | Sony Computer Entertainment Inc. | Inertially trackable hand-held controller |
US7646372B2 (en) * | 2003-09-15 | 2010-01-12 | Sony Computer Entertainment Inc. | Methods and systems for enabling direction detection when interfacing with a computer program |
US7623115B2 (en) | 2002-07-27 | 2009-11-24 | Sony Computer Entertainment Inc. | Method and apparatus for light input device |
US8570378B2 (en) | 2002-07-27 | 2013-10-29 | Sony Computer Entertainment Inc. | Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera |
US9393487B2 (en) | 2002-07-27 | 2016-07-19 | Sony Interactive Entertainment Inc. | Method for mapping movements of a hand-held controller to game commands |
US7760248B2 (en) | 2002-07-27 | 2010-07-20 | Sony Computer Entertainment Inc. | Selective sound source listening in conjunction with computer interactive processing |
US8686939B2 (en) | 2002-07-27 | 2014-04-01 | Sony Computer Entertainment Inc. | System, method, and apparatus for three-dimensional input control |
US8313380B2 (en) | 2002-07-27 | 2012-11-20 | Sony Computer Entertainment America Llc | Scheme for translating movements of a hand-held controller into inputs for a system |
US7782297B2 (en) | 2002-07-27 | 2010-08-24 | Sony Computer Entertainment America Inc. | Method and apparatus for use in determining an activity level of a user in relation to a system |
US8019121B2 (en) * | 2002-07-27 | 2011-09-13 | Sony Computer Entertainment Inc. | Method and system for processing intensity from input devices for interfacing with a computer program |
US9474968B2 (en) | 2002-07-27 | 2016-10-25 | Sony Interactive Entertainment America Llc | Method and system for applying gearing effects to visual tracking |
US9682319B2 (en) | 2002-07-31 | 2017-06-20 | Sony Interactive Entertainment Inc. | Combiner method for altering game gearing |
US9117342B2 (en) | 2004-09-16 | 2015-08-25 | Bally Gaming, Inc. | Networked gaming system communication protocols and methods |
US9082260B2 (en) | 2004-09-16 | 2015-07-14 | Bally Gaming, Inc. | Networked gaming system communication protocols and methods |
US8986122B2 (en) | 2002-09-13 | 2015-03-24 | Bally Gaming, Inc. | Networked gaming system communication protocols and methods |
US8529349B2 (en) | 2004-09-16 | 2013-09-10 | Bally Gaming, Inc. | Networked gaming system communication protocols and methods |
US8568237B2 (en) | 2004-09-16 | 2013-10-29 | Bally Gaming, Inc. | Networked gaming system communication protocols and methods |
US8992326B2 (en) | 2006-09-06 | 2015-03-31 | Bally Gaming, Inc. | Networked gaming system communication protocols and methods |
US9177387B2 (en) | 2003-02-11 | 2015-11-03 | Sony Computer Entertainment Inc. | Method and apparatus for real time motion capture |
US8072470B2 (en) | 2003-05-29 | 2011-12-06 | Sony Computer Entertainment Inc. | System and method for providing a real-time three-dimensional interactive environment |
US10279254B2 (en) | 2005-10-26 | 2019-05-07 | Sony Interactive Entertainment Inc. | Controller having visually trackable object for interfacing with a gaming system |
US8287373B2 (en) * | 2008-12-05 | 2012-10-16 | Sony Computer Entertainment Inc. | Control device for communicating visual information |
US8323106B2 (en) | 2008-05-30 | 2012-12-04 | Sony Computer Entertainment America Llc | Determination of controller three-dimensional location using image analysis and ultrasonic communication |
US9573056B2 (en) | 2005-10-26 | 2017-02-21 | Sony Interactive Entertainment Inc. | Expandable control device via hardware attachment |
US7663689B2 (en) * | 2004-01-16 | 2010-02-16 | Sony Computer Entertainment Inc. | Method and apparatus for optimizing capture device settings through depth information |
US8547401B2 (en) | 2004-08-19 | 2013-10-01 | Sony Computer Entertainment Inc. | Portable augmented reality device and method |
US10803694B2 (en) | 2004-09-16 | 2020-10-13 | Sg Gaming, Inc. | Player gaming console, gaming machine, networked gaming system |
US8678901B1 (en) | 2005-09-07 | 2014-03-25 | Bally Gaming | System gaming |
EP1946203A2 (en) * | 2005-10-26 | 2008-07-23 | Sony Computer Entertainment America, Inc. | System and method for interfacing with a computer program |
US20070174235A1 (en) * | 2006-01-26 | 2007-07-26 | Michael Gordon | Method of using digital characters to compile information |
US7667120B2 (en) * | 2006-03-30 | 2010-02-23 | The Tsi Company | Training method using specific audio patterns and techniques |
GB0613955D0 (en) * | 2006-07-13 | 2007-01-10 | Bae Systems Plc | Controller |
US7702608B1 (en) | 2006-07-14 | 2010-04-20 | Ailive, Inc. | Generating motion recognizers for arbitrary motions for video games and tuning the motion recognizers to the end user |
US8310656B2 (en) | 2006-09-28 | 2012-11-13 | Sony Computer Entertainment America Llc | Mapping movements of a hand-held controller to the two-dimensional image plane of a display screen |
USRE48417E1 (en) | 2006-09-28 | 2021-02-02 | Sony Interactive Entertainment Inc. | Object direction using video input combined with tilt angle information |
US8781151B2 (en) | 2006-09-28 | 2014-07-15 | Sony Computer Entertainment Inc. | Object detection using video input combined with tilt angle information |
US20080096657A1 (en) * | 2006-10-20 | 2008-04-24 | Sony Computer Entertainment America Inc. | Method for aiming and shooting using motion sensing controller |
US7636697B1 (en) | 2007-01-29 | 2009-12-22 | Ailive Inc. | Method and system for rapid evaluation of logical expressions |
US8144148B2 (en) * | 2007-02-08 | 2012-03-27 | Edge 3 Technologies Llc | Method and system for vision-based interaction in a virtual environment |
US20080215994A1 (en) * | 2007-03-01 | 2008-09-04 | Phil Harrison | Virtual world avatar control, interactivity and communication interactive messaging |
US8996409B2 (en) | 2007-06-06 | 2015-03-31 | Sony Computer Entertainment Inc. | Management of online trading services using mediated communications |
US8902227B2 (en) * | 2007-09-10 | 2014-12-02 | Sony Computer Entertainment America Llc | Selective interactive mapping of real-world objects to create interactive virtual-world objects |
US8903756B2 (en) * | 2007-10-19 | 2014-12-02 | Ying Zhao | System and method for knowledge pattern search from networked agents |
US8542907B2 (en) | 2007-12-17 | 2013-09-24 | Sony Computer Entertainment America Llc | Dynamic three-dimensional object mapping for user-defined control device |
CN102016877B (en) | 2008-02-27 | 2014-12-10 | 索尼计算机娱乐美国有限责任公司 | Methods for capturing depth data of a scene and applying computer actions |
US8368753B2 (en) * | 2008-03-17 | 2013-02-05 | Sony Computer Entertainment America Llc | Controller with an integrated depth camera |
US8447421B2 (en) * | 2008-08-19 | 2013-05-21 | Sony Computer Entertainment Inc. | Traffic-based media selection |
US8290604B2 (en) * | 2008-08-19 | 2012-10-16 | Sony Computer Entertainment America Llc | Audience-condition based media selection |
US8221229B2 (en) * | 2008-10-27 | 2012-07-17 | Sony Computer Entertainment Inc. | Spherical ended controller with configurable modes |
US20100241893A1 (en) | 2009-03-18 | 2010-09-23 | Eric Friedman | Interpretation and execution of a customizable database request using an extensible computer process and an available computing environment |
US8527657B2 (en) | 2009-03-20 | 2013-09-03 | Sony Computer Entertainment America Llc | Methods and systems for dynamically adjusting update rates in multi-player network gaming |
US8342963B2 (en) | 2009-04-10 | 2013-01-01 | Sony Computer Entertainment America Inc. | Methods and systems for enabling control of artificial intelligence game characters |
US8142288B2 (en) * | 2009-05-08 | 2012-03-27 | Sony Computer Entertainment America Llc | Base station movement detection and compensation |
US8393964B2 (en) * | 2009-05-08 | 2013-03-12 | Sony Computer Entertainment America Llc | Base station for position location |
US9417700B2 (en) | 2009-05-21 | 2016-08-16 | Edge3 Technologies | Gesture recognition systems and related methods |
US9430521B2 (en) * | 2009-09-30 | 2016-08-30 | Microsoft Technology Licensing, Llc | Query expansion through searching content identifiers |
US8396252B2 (en) | 2010-05-20 | 2013-03-12 | Edge 3 Technologies | Systems and related methods for three dimensional gesture recognition in vehicles |
US8582866B2 (en) | 2011-02-10 | 2013-11-12 | Edge 3 Technologies, Inc. | Method and apparatus for disparity computation in stereo images |
US8655093B2 (en) | 2010-09-02 | 2014-02-18 | Edge 3 Technologies, Inc. | Method and apparatus for performing segmentation of an image |
WO2012030872A1 (en) | 2010-09-02 | 2012-03-08 | Edge3 Technologies Inc. | Method and apparatus for confusion learning |
US8666144B2 (en) | 2010-09-02 | 2014-03-04 | Edge 3 Technologies, Inc. | Method and apparatus for determining disparity of texture |
US8504487B2 (en) | 2010-09-21 | 2013-08-06 | Sony Computer Entertainment America Llc | Evolution of a user interface based on learned idiosyncrasies and collected data of a user |
US8484219B2 (en) | 2010-09-21 | 2013-07-09 | Sony Computer Entertainment America Llc | Developing a knowledge base associated with a user that facilitates evolution of an intelligent user interface |
KR20120047577A (en) * | 2010-11-04 | 2012-05-14 | 주식회사 케이티 | Apparatus and method for providing robot interaction services using interactive behavior model |
US8970589B2 (en) | 2011-02-10 | 2015-03-03 | Edge 3 Technologies, Inc. | Near-touch interaction with a stereo camera grid structured tessellations |
US9672609B1 (en) | 2011-11-11 | 2017-06-06 | Edge 3 Technologies, Inc. | Method and apparatus for improved depth-map estimation |
US8849730B2 (en) * | 2011-12-15 | 2014-09-30 | Microsoft Corporation | Prediction of user response actions to received data |
US9384448B2 (en) * | 2011-12-28 | 2016-07-05 | General Electric Company | Action-based models to identify learned tasks |
US8554705B1 (en) | 2012-10-29 | 2013-10-08 | Wargaming.Net Llp | Computer based education for use with multiuser software |
US9105178B2 (en) | 2012-12-03 | 2015-08-11 | Sony Computer Entertainment Inc. | Remote dynamic configuration of telemetry reporting through regular expressions |
US10721448B2 (en) | 2013-03-15 | 2020-07-21 | Edge 3 Technologies, Inc. | Method and apparatus for adaptive exposure bracketing, segmentation and scene organization |
CN103472756A (en) * | 2013-09-27 | 2013-12-25 | 腾讯科技(深圳)有限公司 | Artificial intelligence achieving method, server and equipment |
US9452363B2 (en) | 2013-10-18 | 2016-09-27 | Sony Interactive Entertainment America Llc | Multi-player video game environment for single-player use |
EP3073439A1 (en) | 2015-03-23 | 2016-09-28 | Thales | Method for determining tactical actions |
ES2692843T3 (en) | 2015-03-23 | 2018-12-05 | Thales | Procedure to assess the level of threat |
US9799326B2 (en) | 2016-01-26 | 2017-10-24 | International Business Machines Corporation | Training a cognitive agent using document output generated from a recorded process |
US10873545B2 (en) * | 2017-06-12 | 2020-12-22 | Microsoft Technology Licensing, Llc | Automatic action responses |
US11556548B2 (en) | 2017-08-08 | 2023-01-17 | Microsoft Technology Licensing, Llc | Intelligent query system for attachments |
US10387161B2 (en) * | 2017-09-01 | 2019-08-20 | Facebook, Inc. | Techniques for capturing state information and performing actions for threads in a multi-threaded computing environment |
CN112753008B (en) * | 2018-09-27 | 2024-04-12 | 苹果公司 | Intermediate emerging content |
US11331581B2 (en) | 2019-03-19 | 2022-05-17 | modl.ai ApS | Experience based game development and methods for use therewith |
US11596867B2 (en) | 2019-03-19 | 2023-03-07 | modl.ai ApS | AI-based content generation for gaming applications |
US10918948B2 (en) | 2019-03-19 | 2021-02-16 | modl.ai ApS | Game bot generation for gaming applications |
US11325048B2 (en) | 2019-03-19 | 2022-05-10 | modl.ai ApS | User experience modeling for gaming applications |
JP2022536931A (en) * | 2019-06-14 | 2022-08-22 | ファイター・ベース・パブリッシング,インコーポレイテッド | Method and system for artificial intelligence powered user interface |
CN111632379B (en) * | 2020-04-28 | 2022-03-22 | 腾讯科技(深圳)有限公司 | Game role behavior control method and device, storage medium and electronic equipment |
US11534690B2 (en) * | 2020-08-21 | 2022-12-27 | Electronic Arts Inc. | Readable and editable NPC behavior creation using reinforcement learning |
US20220164702A1 (en) * | 2020-11-20 | 2022-05-26 | SimpleBet, Inc. | System, method, and model structure for using machine learning to predict future sport outcomes based on match state transitions |
US11794107B2 (en) * | 2020-12-30 | 2023-10-24 | Activision Publishing, Inc. | Systems and methods for improved collision detection in video games |
WO2023027922A1 (en) * | 2021-08-27 | 2023-03-02 | modl.ai ApS | Quality assurance game bots for gaming applications |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6072466A (en) * | 1996-08-02 | 2000-06-06 | U.S. Philips Corporation | Virtual environment manipulation device modelling and control |
US6201948B1 (en) * | 1996-05-22 | 2001-03-13 | Netsage Corporation | Agent based instruction system and method |
US6427063B1 (en) * | 1997-05-22 | 2002-07-30 | Finali Corporation | Agent based instruction system and method |
US6425582B1 (en) * | 2000-07-21 | 2002-07-30 | Joe Rosi | Gangster board game |
US6561811B2 (en) * | 1999-08-09 | 2003-05-13 | Entertainment Science, Inc. | Drug abuse prevention computer game |
US6892349B2 (en) * | 2000-09-07 | 2005-05-10 | Sony Corporation | Information processing apparatus, method and program for displaying one or more characters on a plurality of application display areas responsive to scenario information |
US7054928B2 (en) * | 1999-12-23 | 2006-05-30 | M.H. Segan Limited Partnership | System for viewing content over a network and method therefor |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2897984B1 (en) * | 1998-06-03 | 1999-05-31 | コナミ株式会社 | VIDEO GAME DEVICE, METHOD OF GUIDING SPECIFICATION OF CHARACTER POSITION, AND READABLE RECORDING MEDIUM RECORDING GAME PROGRAM TO GUIDE SPECIFICATION OF CHARACTER POSITION |
US6424356B2 (en) * | 1999-05-05 | 2002-07-23 | Immersion Corporation | Command of force sensations in a forceback system using force effect suites |
US6709335B2 (en) * | 2001-09-19 | 2004-03-23 | Zoesis, Inc. | Method of displaying message in an interactive computer process during the times of heightened user interest |
US7296007B1 (en) | 2004-07-06 | 2007-11-13 | Ailive, Inc. | Real time context learning by software agents |
-
2004
- 2004-07-06 US US10/885,495 patent/US7296007B1/en not_active Expired - Fee Related
-
2005
- 2005-07-06 WO PCT/US2005/024139 patent/WO2006014560A2/en active Application Filing
-
2007
- 2007-10-30 US US11/929,170 patent/US7636701B2/en not_active Expired - Fee Related
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6201948B1 (en) * | 1996-05-22 | 2001-03-13 | Netsage Corporation | Agent based instruction system and method |
US6072466A (en) * | 1996-08-02 | 2000-06-06 | U.S. Philips Corporation | Virtual environment manipulation device modelling and control |
US6427063B1 (en) * | 1997-05-22 | 2002-07-30 | Finali Corporation | Agent based instruction system and method |
US6561811B2 (en) * | 1999-08-09 | 2003-05-13 | Entertainment Science, Inc. | Drug abuse prevention computer game |
US7054928B2 (en) * | 1999-12-23 | 2006-05-30 | M.H. Segan Limited Partnership | System for viewing content over a network and method therefor |
US6425582B1 (en) * | 2000-07-21 | 2002-07-30 | Joe Rosi | Gangster board game |
US6892349B2 (en) * | 2000-09-07 | 2005-05-10 | Sony Corporation | Information processing apparatus, method and program for displaying one or more characters on a plurality of application display areas responsive to scenario information |
Cited By (79)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7636701B2 (en) | 2004-07-06 | 2009-12-22 | Ailive, Inc. | Query controlled behavior models as components of intelligent agents |
US20080097948A1 (en) * | 2004-07-06 | 2008-04-24 | Ailive, Inc. | Real Time Context Learning by Software Agents |
US20060200588A1 (en) * | 2005-02-04 | 2006-09-07 | Samsung Electronics Co., Ltd. | Method and apparatus for determining load of input/output command in input/output subsystem |
US20070136745A1 (en) * | 2005-12-08 | 2007-06-14 | International Business Machines Corporation | Brokering of personalized rulesets for use in digital media character replacement |
US20070132780A1 (en) * | 2005-12-08 | 2007-06-14 | International Business Machines Corporation | Control of digital media character replacement using personalized rulesets |
US8683333B2 (en) * | 2005-12-08 | 2014-03-25 | International Business Machines Corporation | Brokering of personalized rulesets for use in digital media character replacement |
US9471924B2 (en) | 2005-12-08 | 2016-10-18 | International Business Machines Corporation | Control of digital media character replacement using personalized rulesets |
US20080115086A1 (en) * | 2006-11-15 | 2008-05-15 | Yahoo! Inc. | System and method for recognizing and storing information and associated context |
US20080115149A1 (en) * | 2006-11-15 | 2008-05-15 | Yahoo! Inc. | System and method for providing context information |
US20080114758A1 (en) * | 2006-11-15 | 2008-05-15 | Yahoo! Inc. | System and method for information retrieval using context information |
US8005806B2 (en) | 2006-11-15 | 2011-08-23 | Yahoo! Inc. | System and method for information retrieval using context information |
US8522257B2 (en) | 2006-11-15 | 2013-08-27 | Yahoo! Inc. | System and method for context information retrieval |
US8056007B2 (en) * | 2006-11-15 | 2011-11-08 | Yahoo! Inc. | System and method for recognizing and storing information and associated context |
US20080154912A1 (en) * | 2006-12-22 | 2008-06-26 | Yahoo! Inc. | Method and system for locating events in-context |
US8185524B2 (en) | 2006-12-22 | 2012-05-22 | Yahoo! Inc. | Method and system for locating events in-context |
US7636645B1 (en) | 2007-06-18 | 2009-12-22 | Ailive Inc. | Self-contained inertial navigation system for interactive control using movable controllers |
US20090307671A1 (en) * | 2008-06-06 | 2009-12-10 | Cornell University | System and method for scaling simulations and games |
US8443350B2 (en) * | 2008-06-06 | 2013-05-14 | Cornell University | System and method for scaling simulations and games |
US8554526B2 (en) | 2008-07-22 | 2013-10-08 | Sony Online Entertainment Llc | System and method for physics interactions in a simulation |
US20100029384A1 (en) * | 2008-07-22 | 2010-02-04 | Sony Online Entertainment Llc | System and method for physics interactions in a simulation |
KR101357978B1 (en) | 2008-07-22 | 2014-02-03 | 소니 온라인 엔터테인먼트 엘엘씨 | System and method for physics interactions in a simulation |
WO2010011770A3 (en) * | 2008-07-22 | 2010-04-15 | Sony Online Entertainment Llc | System and method for physics interactions in a simulation |
US9202171B2 (en) * | 2008-11-11 | 2015-12-01 | Digideal Corporation | Virtual game assistant based on artificial intelligence |
US20100121808A1 (en) * | 2008-11-11 | 2010-05-13 | Kuhn Michael J | Virtual game dealer based on artificial intelligence |
US20110250575A1 (en) * | 2010-04-13 | 2011-10-13 | enVie Interactive LLC | System And Method For Providing A Visual Representation Of A User Personality Within A Virtual Environment |
US9033796B2 (en) | 2011-01-12 | 2015-05-19 | Kabushiki Kaisha Square Enix | Automatic movement of player character in network game |
US11103791B2 (en) | 2011-01-12 | 2021-08-31 | Kabushiki Kaisha Square Enix | Automatic movement of player character in network game |
US9308458B2 (en) | 2011-01-12 | 2016-04-12 | Kabushiki Kaisha Square Enix | Automatic movement of player character in network game |
US8821290B2 (en) | 2011-01-12 | 2014-09-02 | Kabushiki Kaisha Square Enix | Automatic movement of disconnected character in network game |
US9975049B2 (en) | 2011-01-12 | 2018-05-22 | Kabushiki Kaisha Square Enix | Automatic movement of player character in network game |
US10569178B2 (en) | 2011-01-12 | 2020-02-25 | Kabushiki Kaisha Square Enix | Automatic movement of player character in network game |
US10788949B2 (en) * | 2013-07-19 | 2020-09-29 | Kabam Inc. | Facilitating automatic execution of user interactions in a virtual space |
US11086477B2 (en) | 2013-07-19 | 2021-08-10 | Kabam, Inc. | Facilitating automatic execution of user interactions in a virtual space |
US10394414B1 (en) * | 2013-07-19 | 2019-08-27 | Kabam, Inc. | Facilitating automatic execution of user interactions in a virtual space |
US20150051000A1 (en) * | 2013-08-16 | 2015-02-19 | Disney Enterprises, Inc. | Perception system |
US9586140B2 (en) * | 2013-08-16 | 2017-03-07 | Disney Enterprises, Inc. | Perception system |
US10188950B2 (en) * | 2014-10-23 | 2019-01-29 | Nokia Technologies Oy | Method and apparatus for providing privacy policy generation based on in-game behavior data |
US20160114247A1 (en) * | 2014-10-23 | 2016-04-28 | Here Global B.V. | Method and apparatus for providing privacy policy generation based on in-game behavior data |
US10042514B2 (en) * | 2014-10-30 | 2018-08-07 | Microsoft Technology Licensing, Llc | Typeahead features |
US11524237B2 (en) | 2015-05-14 | 2022-12-13 | Activision Publishing, Inc. | Systems and methods for distributing the generation of nonplayer characters across networked end user devices for use in simulated NPC gameplay sessions |
US11896905B2 (en) | 2015-05-14 | 2024-02-13 | Activision Publishing, Inc. | Methods and systems for continuing to execute a simulation after processing resources go offline |
US10933322B2 (en) * | 2015-06-30 | 2021-03-02 | FanDuel Limited | Method and system for encouraging a person to affirmatively engage in an activity |
US10300379B1 (en) | 2015-12-28 | 2019-05-28 | Electronic Arts Inc. | Facilitating contextual game notifications and smart action options |
US10286322B1 (en) | 2016-01-25 | 2019-05-14 | Electronic Arts Inc. | System and method for determining and executing actions in an online game |
US11369880B2 (en) | 2016-03-08 | 2022-06-28 | Electronic Arts Inc. | Dynamic difficulty adjustment |
US10810791B2 (en) * | 2016-04-28 | 2020-10-20 | Verizon Patent And Licensing Inc. | Methods and systems for distinguishing objects in a natural setting to create an individually-manipulable volumetric model of an object |
US20190156565A1 (en) * | 2016-04-28 | 2019-05-23 | Verizon Patent And Licensing Inc. | Methods and Systems for Distinguishing Objects in a Natural Setting to Create an Individually-Manipulable Volumetric Model of an Object |
US11458399B2 (en) | 2016-12-30 | 2022-10-04 | Electronic Arts Inc. | Systems and methods for automatically measuring a video game difficulty |
US11413539B2 (en) | 2017-02-28 | 2022-08-16 | Electronic Arts Inc. | Realtime dynamic modification and optimization of gameplay parameters within a video game application |
US11511194B2 (en) | 2017-03-07 | 2022-11-29 | Sony Interactive Entertainment LLC | Emulating player behavior after player departure |
US10512846B2 (en) * | 2017-03-07 | 2019-12-24 | Sony Interactive Entertainment LLC | Emulating player behavior after player departure |
US10946280B2 (en) | 2017-03-08 | 2021-03-16 | Sony Interactive Entertainment LLC | In-game reactions to interruptions |
US11763143B2 (en) | 2017-04-19 | 2023-09-19 | AIBrain Corporation | Adding deep learning based AI control |
US20180314963A1 (en) * | 2017-04-19 | 2018-11-01 | AIBrain Corporation | Domain-independent and scalable automated planning system using deep neural networks |
US11413536B2 (en) | 2017-12-22 | 2022-08-16 | Activision Publishing, Inc. | Systems and methods for managing virtual items across multiple video game environments |
US10812510B2 (en) * | 2018-01-12 | 2020-10-20 | The Boeing Company | Anticipatory cyber defense |
US20210027164A1 (en) * | 2018-01-22 | 2021-01-28 | Apple Inc. | Objective-effectuators in synthesized reality settings |
CN112218693A (en) * | 2018-03-23 | 2021-01-12 | 索尼互动娱乐有限责任公司 | Voice help system using artificial intelligence |
US11532172B2 (en) | 2018-06-13 | 2022-12-20 | Electronic Arts Inc. | Enhanced training of machine learning systems based on automatically generated realistic gameplay information |
US11878238B2 (en) | 2018-06-14 | 2024-01-23 | Sony Interactive Entertainment Inc. | System and method for generating an input signal |
GB2574650A (en) * | 2018-06-14 | 2019-12-18 | Sony Interactive Entertainment Inc | System and method for generating an input signal |
US11478713B1 (en) | 2018-09-13 | 2022-10-25 | Electronic Arts Inc. | Automated detection of emergent behaviors in interactive agents of an interactive environment |
US10881964B1 (en) * | 2018-09-13 | 2021-01-05 | Electronic Arts Inc. | Automated detection of emergent behaviors in interactive agents of an interactive environment |
US11679330B2 (en) * | 2018-12-18 | 2023-06-20 | Activision Publishing, Inc. | Systems and methods for generating improved non-player characters |
US20200197811A1 (en) * | 2018-12-18 | 2020-06-25 | Activision Publishing, Inc. | Systems and Methods for Generating Improved Non-Player Characters |
US11276216B2 (en) | 2019-03-27 | 2022-03-15 | Electronic Arts Inc. | Virtual animal character generation from image or video data |
US11406899B2 (en) | 2019-03-27 | 2022-08-09 | Electronic Arts Inc. | Virtual character generation from image or video data |
US10946281B2 (en) * | 2019-03-29 | 2021-03-16 | Nvidia Corporation | Using playstyle patterns to generate virtual representations of game players |
US11944903B2 (en) | 2019-03-29 | 2024-04-02 | Nvidia Corporation | Using playstyle patterns to generate virtual representations of game players |
CN110339569A (en) * | 2019-07-08 | 2019-10-18 | 深圳市腾讯网域计算机网络有限公司 | Control the method and device of virtual role in scene of game |
US11110353B2 (en) * | 2019-07-10 | 2021-09-07 | Electronic Arts Inc. | Distributed training for machine learning of AI controlled virtual entities on video game clients |
US11712627B2 (en) | 2019-11-08 | 2023-08-01 | Activision Publishing, Inc. | System and method for providing conditional access to virtual gaming items |
US11524234B2 (en) | 2020-08-18 | 2022-12-13 | Activision Publishing, Inc. | Multiplayer video games with virtual characters having dynamically modified fields of view |
US11351459B2 (en) | 2020-08-18 | 2022-06-07 | Activision Publishing, Inc. | Multiplayer video games with virtual characters having dynamically generated attribute profiles unconstrained by predefined discrete values |
US20230034222A1 (en) * | 2021-07-28 | 2023-02-02 | Blizzard Entertainment, Inc. | Initial results of a reinforcement learning model using a heuristic |
US11724194B2 (en) * | 2021-07-28 | 2023-08-15 | Blizzard Entertainment, Inc. | Initial results of a reinforcement learning model using a heuristic |
CN113648660A (en) * | 2021-08-16 | 2021-11-16 | 网易(杭州)网络有限公司 | Behavior sequence generation method and device for non-player character |
WO2023039564A1 (en) * | 2021-09-12 | 2023-03-16 | Sony Interactive Entertainment Inc. | Using data from a game metadata system to create actionable in-game decisions |
US20230082732A1 (en) * | 2021-09-12 | 2023-03-16 | Sony Interactive Entertainment Inc. | Using data from a game metadata system to create actionable in-game decisions |
Also Published As
Publication number | Publication date |
---|---|
US7296007B1 (en) | 2007-11-13 |
WO2006014560A3 (en) | 2009-04-09 |
US7636701B2 (en) | 2009-12-22 |
WO2006014560A2 (en) | 2006-02-09 |
US20080097948A1 (en) | 2008-04-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7296007B1 (en) | Real time context learning by software agents | |
Yannakakis et al. | Artificial intelligence and games | |
Risi et al. | From chess and atari to starcraft and beyond: How game ai is driving the world of ai | |
Schwab et al. | AI game engine programming | |
Laird | Using a computer game to develop advanced AI | |
Togelius | Playing smart: On games, intelligence, and artificial intelligence | |
Merrick | Computational models of motivation for game-playing agents | |
Togelius | Optimization, imitation and innovation: Computational intelligence and games | |
Zook et al. | Temporal game challenge tailoring | |
Khatri | The Gaming Experience With AI | |
Mahlmann | Modeling and Generating Strategy Games Mechanics | |
Yannakakis et al. | Playing Games | |
Patel et al. | Improving behavior of computer game bots using fictitious play | |
Alves | Modeling of synthetic players as an instrument for testing generative content | |
Nielsen et al. | Ai for general strategy game playing | |
Ferguson | Machine Learning Arena | |
Bakkes | Rapid adaptation of video game AI | |
Mozgovoy et al. | Behavior capture: Building believable and effective AI agents for video games | |
US20240050858A1 (en) | Automated Validation of Video Game Environments | |
Reis | Artificial Intelligence Methods for Automated Difficulty and Power Balance in Games | |
Armanto et al. | Evolutionary Algorithm in Game–A Systematic Review | |
Jones | Benefits of genetic algorithms in simulations for game designers | |
dos Santos Lima | Adaptivity in Single Player Video Games | |
Christos | Behavior of Intelligent Agents Using Reinforcement Learning Through Unity's ML-Agent’s Toolkit | |
Glavin | Towards inherently adaptive first person shooter agents using reinforcement learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: IKUNI, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FUNGE, JOHN;MUSICK, RON;DOBSON, DANIEL;AND OTHERS;REEL/FRAME:015169/0538;SIGNING DATES FROM 20040907 TO 20040921 |
|
AS | Assignment |
Owner name: AILIVE INC., CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:IKUNI INC.;REEL/FRAME:018117/0073 Effective date: 20060426 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: PAT HOLDER CLAIMS SMALL ENTITY STATUS, ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: LTOS); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
AS | Assignment |
Owner name: AILIVE HOLDING CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AILIVE INC.;REEL/FRAME:042802/0167 Effective date: 20170622 Owner name: YEN, WEI, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AILIVE HOLDING CORPORATION;REEL/FRAME:042803/0830 Effective date: 20170622 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20191113 |