US20090177321A1 - Robot for participating in a joint performance with a human partner - Google Patents
Robot for participating in a joint performance with a human partner Download PDFInfo
- Publication number
- US20090177321A1 US20090177321A1 US12/401,873 US40187309A US2009177321A1 US 20090177321 A1 US20090177321 A1 US 20090177321A1 US 40187309 A US40187309 A US 40187309A US 2009177321 A1 US2009177321 A1 US 2009177321A1
- Authority
- US
- United States
- Prior art keywords
- robot
- partner
- behavior
- recognized
- analyzer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/008—Artificial life, i.e. computing arrangements simulating life based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
Definitions
- the present invention relates generally to robots, and more specifically to a robot for participating in a joint performance with a human partner and a method of controlling a robot.
- the robot is useful for jointly performing a comic dialogue with a comedian partner in a theater and for assisting a schoolteacher in a classroom.
- robots which are described in Japanese Patent Publication 2000-353012 for nursing the aged or the handicapped.
- the robot has an eye whose viewing point is determined with respect to the position of a user.
- the robot interprets the voice command at the determined viewing point and determines its direction of motion.
- the known robots are capable of recognizing human voice and of using light and sound to express their own feelings, their performance is limited to responding to only a simple phrase or action by a user. Therefore, their performance is a series of mutually unrelated behaviors.
- a robot comprising a first analyzer for analyzing phrase and action of a human partner to detect a recognized behavior of the partner, a second analyzer for analyzing a state of a plurality of people listening to utterances from the partner and the robot to detect a recognized state of the people, a scenario memory for storing a scenario describing a dialogue between the partner and the robot, and a processor fox making reference to a portion of the scenario in the memory according to one of the recognized behavior of the partner and the recognized state of the people, and determining a behavior of the robot according to the referenced portion of the scenario.
- the present invention provides a robot control system for controlling a robot, comprising a first analyzer for analyzing phrase and action of a human partner to detect a recognized behavior of the partner, a second analyzer for analyzing a state of a plurality of people listening to utterances from the partner and the robot to detect a recognized state of the people, a scenario memory for storing a scenario describing a dialogue between the partner and the robot, and a processor for making reference to a portion of the scenario in the memory according to one of the recognized behavior of the partner and the recognized state of the people, and determining a behavior of the robot according to the referenced portion of the scenario.
- the present invention provides a method of controlling a robot, comprising the steps of (a) storing a scenario describing a dialogue between a human partner and the robot in a memory, (b) analyzing phrase and action of a human partner to detect a recognized behavior of the partner and analyzing a state of a plurality of people listening to utterances from the partner and the robot to detect a recognized state of the people, and (c) making reference to a portion of the scenario in the memory according to at least one of the recognized behavior of the partner and the recognized state of the people and determining a behavior of the robot according to the referenced portion of the scenario.
- the present invention provides a computer-readable storage medium containing a stored scenario describing a dialogue between a human partner and a robot and a program for controlling the robot, the program performing the steps of (a) analyzing phrase and action of a human partner to detect a recognized behavior of the partner and analyzing a state of a plurality of people listening to utterances from the partner and the robot to detect a recognized state of the people, and (b) making reference to a portion of the stored scenario according to at least one of the recognized behavior of the partner and the recognized state of the people and determining a behavior of the robot according to the referenced portion of the scenario.
- FIGS. 1A and 1B are front and side views of an entertainment robot of the present invention, respectively;
- FIG. 2 is a block diagram of the robot of according to a first embodiment of the present invention, in which the robot ads as an artificial partner for jointly performing a comic dialogue with a human comedian partner;
- FIG. 3 is an illustration of the scenario memory of FIG. 2 according to the first embodiment of the invention.
- FIG. 4 is an illustration of the action pattern memory of the robot behavior database of FIG. 2 for comical actions to be performed by the robot;
- FIG. 5 is an illustration of the phrase pattern memory of the robot behavior database for showing comical phrases to be uttered by the robot;
- FIG. 6 is a flowchart of the operation of the processor of FIG. 2 according to the first embodiment
- FIG. 7 is an illustration of a modified scenario memory according to a modification of the first embodiment of this invention.
- FIG. 8 is a flowchart of the operation of the processor according to the modification of the invention.
- FIG. 9 is a block diagram of the robot according to a farther modification of the first embodiment of the invention.
- FIG. 10 is a block diagram of a robot according to a second embodiment of the present invention, in which the robot acts as an artificial assistance for a schoolteacher in a classroom for teaching a group of pupils.
- an entertainment robot is shown in FIGS. 1A and 1B .
- the entertainment robot is used on the stage of a hall or theater as an artificial comedian, standing on the side of a human comedian partner. They perform a comic dialogue by exchanging comical phrases with each other on a wide range of topics including social, political and human affairs.
- the entertainment robot comprises a torso 1 of generally cylindrical structure with a rounded upper portion 2 having a circular opening 3 and a pair of limb portions 4 on its front Bide.
- a head 5 of spherical shape is partially submerged in the opening 3 and rotatably secured so that it can orient its face in a limited horizontal angle about a vertical axis 5 A and in a limited vertical angle about a horizontal axis 5 B.
- Each of the limb portions 4 extends outwards from the torso 1 and includes a driven wheel 6 .
- a castor 7 is fitted to the bottom of torso 1 near its rear side so that the robot is capable of moving about on the stage.
- Head 5 is provided with a CCD camera 8 A in the position of an eye, a thermographic sensor 8 B in the position of the other eye and a touch sensor 10 on top of the head.
- a microphone 9 and a loudspeaker 11 are provided on the front side of torso 1 .
- the outputs of CCD camera 8 A, thermographic sensor 8 B, microphone 9 and touch sensor 10 are connected to a partner's behavior analyzer 12 .
- Partner's behavior analyzer 12 uses its input signals to determine what the partner has behaved by consulting a partner's feature memory 13 in which a plurality of facial features and speech patterns of the partner are stored as references.
- Analyzer 12 uses the thermographic sensor 8 B to analyze the feeling of the partner to detect a recognized partner's behavior.
- CCD camera 8 A and microphone 9 are coupled to an audience state analyzer 14 , which determines whether the theater audience is large or small and analyzes their reaction in response to amusing phrases or serious remarks from the performers.
- Robot behavior database 16 includes an action pattern memory 16 A and a phrase pattern memory 16 B. The details of these memories will be described below.
- the robot includes a pair of head actuators 21 and 22 for respectively controlling the orientation of the head 5 in vertical and horizontal angular positions and a pair of wheel actuators 23 and 24 for moving the robot in a specified direction.
- Controller 20 responds to the output signal from the processor 17 to individually control these actuators.
- the scenario memory 15 is divided into a plurality of entries (rows) numbered and arranged according to the scenario of a comic dialogue being played by the comedian partner and the robot. Each entry is sub-divided into a plurality of fields (columns) 31 to 34 .
- a specified partner's action or phrase is indicated in the field 31 , and one or more database keys (A 001 and B 001 , for example) are indicated in the field 32 for making a search through the robot database 16 for corresponding robot action and phrase.
- the field 33 of each entry is a right-to-speak field that indicates which of the comedian or robot has the right to speak first in that entry.
- the field 34 is an audience field for which the robot is programmed to consider an expected scale of audience (large or small) and an expected reaction of audience (laughter or no laughter). The symbol “x” in the audience field 34 indicates that there is no need for the robot to consider audience reaction.
- Robot behavior field 32 of entries numbered 3 , 5 , 6 , and 10 is further sub-divided into on upper sub-field and a lower sub-field.
- the lower sub-field specifies an alternative robot's behavior to be performed when the partner fails to do according to the scenario specified in the partner's behavior field 31 .
- the lower sub-field specifies an alternative robot's behavior to be performed when sensed audience state (scale or reaction) does not match the expected audience state specified in the audience field 34 .
- the action pattern memory 16 A of robot behavior database 16 has a plurality of entries (rows) in which database keys A 001 through A 008 are mapped to corresponding patterns of robot action and sensed partner's behaviors, which will be detected by the analyzer 12 .
- the action pattern memory 16 A is searched for a corresponding pattern of action, which the robot is programmed to perform.
- the robot actions may include bowing, turning around, stepping forward and combinations of those actions.
- the phrase pattern memory 16 B of robot behavior database 16 has a plurality of entries (rows) in which database keys B 001 through B 008 are mapped to corresponding phrase patterns of the robot and sensed partner's behaviors, which will be detected by the analyzer 12 .
- the phrase pattern memory 16 B is searched for a corresponding phrase pattern, which the robot is programmed to utter.
- FIG. 3 shows phrase keys B 001 to B 007 of FIG. 4 .
- action keys specified in FIG. 5 may be combined with the phrase keys to indicate a desired robot behavior.
- processor 17 proceeds according to the flowchart of FIG. 6 .
- Processor 17 initially sets a pointer of the scenario memory 15 to the first entry and begins with step 101 by examining the right-to-speak field 33 of the entry of the scenario memory 15 indicated by the pointer.
- the processor 17 determines which side has the right to speak, the comedian partner ox the robot. If the partner has the right to speak, flow proceeds from step 102 to step 103 to examine the audience field 34 of the current entry and flow proceeds to step 104 to determine whether it is necessary to consider the audience reaction or state.
- step 105 compare a partner's behavior (uttered phrase) detected by the analyzer 12 to the phrase specified in the partner's behavior field 31 of the current entry to detect a match (step 106 ). If they match, flow proceeds from step 106 to step 121 to check to see if the current entry is the last entry. If not, flow proceeds to step 122 to advance the pointer to the next entry. If the decision at step 106 is negative, flow proceeds to step 107 to read a robot behavior data from the lower sub-field of robot's behavior field 32 .
- step 106 If the partner fails to behave according to the scenario specified in the partner's behavior field 31 of the current entry, the decision at step 106 is negative and robot behavior data is read from the lower sub-field of robot behavior field 32 for urging the partner to perform the specified behavior. If the partner has failed to perform the specified behavior a predetermined number of times (step 108 ), flow proceeds to step 121 . Otherwise, flow proceeds to step 109 to perform the specified behavior, and flow returns to step 105 to repeat the process.
- step 104 If the audience field 34 , examined at step 103 , contains an expected audience state, the decision at step 104 is negative and flow proceeds to step 111 to turn the robot face towards the audience to sense the audience state by the audience state analyzer 14 and compares the sensed audience to the expected audience state for a match (step 112 ). If they match, flow proceeds to step 121 . Otherwise, flow proceeds to step 113 to turn the robot face toward the partner to sense his behavior by the partner's behavior analyzer 12 and makes a search through the robot behavior database 16 for a corresponding robot behavior to be performed, using the sensed partner's behavior as a key. If a corresponding robot behavior is specified in the database 16 (step 114 ), flow proceeds to step 120 to perform a robot behavior specified in the robot behavior field 32 (or robot behavior database 16 ). Otherwise, flow proceeds to step 121 .
- step 101 If the right-to-speak field 33 , referenced at step 101 , indicates that the robot has the right to speak first in the current entry, flow proceeds from step 102 to step 115 to examine the audience field 34 . If the audience field 34 contains an expected audience state, the decision at step 116 is negative and flow proceeds to step 117 to turn the robot face toward the audience to detect the audience state by the analyzer 14 and compares the sensed audience state to the expected state for a match (step 118 ). If they match, flow proceeds to step 120 . Otherwise, flow proceeds to step 119 to read a key from the lower sub-field of the robot behavior field 32 . Flow proceeds to step 120 to perform the robot behavior specified in the database 16 corresponding to the retrieved key.
- the processor initializes the address pointer to “1” and examines the right-to-speak field 33 of entry # 1 of the scenario memory 15 (step 101 ) and proceeds to step 102 .
- the right to speak is on the partner side and the robot's human partner is supposed to utter a self-introduction phrase as indicated in the partner's behavior field 31 and the processor proceeds to steps 103 , 104 to examine the audience field 34 .
- step 105 to turn the robot face toward the partner to sense his behavior (i.e., the uttered self-introduction phrase) by the analyzer 12 and compares the partner's behavior to the phrase specified in the partner's behavior field 31 for a match. If they match, flow proceeds through steps 121 and 122 and returns to step 101 to address the entry # 2 of the scenario memory. Therefore, the processor performs the scenario of entry # 1 by following steps 101 - 106 , 121 and 122 (as indicated as pass 1 in FIG. 3 ).
- step 102 the right to speak is on the robot aide (steps 101 , 102 ).
- the processor proceeds from step 102 to step 115 to examine the audience field 34 . Since it is marked with the symbol “yd”, the decision at step 116 is affirmative and the processor proceeds to step 120 to perform the robot behavior specified in the robot behavior field 32 . Since the key B 001 is marked in the robot behavior field 32 , the robot is instructed to utter a self-introduction phrase such as “My name is PAPERO”, and than proceeds through steps 121 and 122 , incrementing the address pointer to “3” before returning to step 101 . As a result, the processor performs the entry # 2 by following steps 101 , 102 , 115 , 116 , 120 - 122 (as indicated as pass 2 in FIG. 3 ).
- the right to speak is on the partner side (steps 101 , 102 ) and the partner is supposed to perform the behavior specified in the field 31 .
- the partner is supposed to slap the robot on the head (i.e., the touch sensor 10 ).
- the processor proceeds to steps 103 , 104 to examine the audience field 34 . Since the audience field of entry # 3 is marked with the symbol “x”, flow proceeds to step 105 to compare the partner's behavior (i.e., the slapping on the robot's head) to the action specified in the partner's behavior field 31 for a match (step 106 ). If they match, flow proceeds from step 106 to step 122 via step 121 to increment the address pointer by one and returns to step 101 .
- the processor follows the pass 1 as in the case of entry # 1 if the partner's behavior matches the specified scenario. If the partner falls to make a slapping action, the decision at step 106 is negative and the processor reads an urging phrase (“Slap My Head”) from the database 16 corresponding to the key B 002 specified in the robot's behavior field 32 (step 107 ) and flow proceeds through step 108 to step 109 to utter the retrieved urging phrase, and flow returns to step 105 . As a result, when the partner fails to behave according to the specified scenario the processor performs steps 101 - 109 , 121 and 122 (pass 3 ).
- step 106 compare the partner's utterance (i.e., the greeting phrase) to the phrase specified in the partner's behavior field 31 for a match (step 106 ). If they match, flow proceeds from step 106 to step 122 via step 121 to increment the address pointer to “5” and returns to step 101 .
- the processor performs steps 101 - 109 , 121 and 122 (pass 3 ) to read an urging phrase from the database 16 corresponding to the key B 003 specified in the robot's behavior field 32 of entry # 3 to urge the robot's partner to utter the missing phrase.
- step 102 the right to speak is on the robot side (steps 101 , 102 ).
- the processor proceeds from step 102 to step 115 to examine the audience field 34 , which contains an expected state of audience (large audience). The decision at step 116 is therefore negative and flow proceeds to step 117 to compare the sensed audience size (i.e., the number of people) to the expected large audience for a match (step 118 ). If they match, flow proceeds to step 120 to read a reactive phrase (“Hello, Everybody”) from the database 16 corresponding to the key B 004 specified in the upper sub-field of robot behavior field 32 and utter the retrieved phrase.
- step 122 the processor increments the address pointer to “6” before returning to step 101 .
- steps 101 , 102 , 115 - 118 , 120 - 122 are executed.
- step 118 If the sensed audience is small, the decision at step 118 is negative and flow proceeds to step 119 to make a search through the database 16 for one of the reactive behaviors using the output of the audience state analyzer 14 and performs the detected reactive behavior (step 120 ). In this case, steps 101 , 102 , 115 - 122 (pass 5 ) are executed.
- step 104 the right to speak is on the partner side (steps 101 , 102 ) and the partner is supposed to utter an opening phrase such as “Today, We Are Going to Perform a Comic Dialogue!”
- the processor proceeds to steps 103 , 104 to examine the audience field 34 , which indicates large audience. Since the decision at step 104 is negative, flow proceeds to step 111 to compare the sensed audience state to the expected large audience for a match (step 112 ). If they match, flow proceeds through step 121 to step 122 to increment the address counter to “7”.
- step 112 If the comparison at step 111 indicates that the sensed audience is small, the decision at step 112 is negative and flow proceeds to step 113 to make a search through the database 16 for a reactive robot behavior using the partner's behavior currently sensed by the partner's behavior analyzer 12 as a key. If a corresponding reactive robot behavior is detected (step 114 ), flow proceeds to step 120 to perform the detected behavior.
- step 122 the processor increments the address pointer to “8” before returning to step 101 .
- steps 101 , 102 , 115 , 116 , 120 - 122 are executed.
- step 102 the right to speak is again on the robot side (steps 101 , 102 ).
- the processor proceeds from step 102 to step 115 to examine the audience field 34 , which contains a “laughter” mark as an expected audience reaction.
- the decision at step 116 is negative and flow proceeds to step 117 to compare the output of the audience state analyzer 14 to the expected audience reaction for a match (step 118 ). If the robot's comical phrase of entry # 7 has appealed to the audience, causing them to laugh, the decision at step 118 is affirmative, and flow proceeds to step 120 to read a reactive phrase from the database 16 corresponding to the key B 006 specified in the robot behavior field 32 and utters the retrieved comical phrase.
- steps 101 , 102 , 115 - 118 , 120 - 122 are executed, incrementing the address pointer to “9”.
- step 118 for the current entry # 8 is negative, and flow proceeds to step 119 to read reactive motion data from the database 16 corresponding to the audience reaction sensed by the audience state analyzer 14 and performs the retrieved reactive motion (step 120 ).
- steps 101 , 102 , 115 - 122 are executed.
- the right to speak is on the partner side (steps 101 , 102 ) and the partner is supposed to utter a comical phrase.
- the processor proceeds to steps 103 , 104 to examine the audience field 34 , which indicates a laughter mark. Since the decision at step 104 is negative, flow proceeds to step 111 to compare the sensed audience state to the expected laughing state of the audience for a match (step 112 ). If the partner's comical phrase during the current entry is appealing to the audience, the decision at step 112 is affirmative and flow proceeds through step 121 to step 122 to increment the address pointer to the next.
- step 112 If the current partner's comical phrase is not appealing to the audience, the decision at step 112 is negative and flow proceeds to step 113 to make a search through the database 16 for a reactive robot behavior using the partner's behavior currently sensed by the partner's behavior analyzer 12 as a key. If a corresponding reactive robot behavior is detected (step 114 ), flow proceeds to step 120 to perform the detected behavior.
- Scenario memory 15 is modified to include further entries # 11 , # 12 , and # 13 as shown in FIG. 7 and the flowchart of FIG. 6 is modified as shown in FIG. 8 .
- the audience held 34 of each entry further includes a symbol “o” which indicates that the processor 17 should recognize the analyzed audience state. Depending on the recognized state, the processor determines its reactive phrase or motion.
- the operation of the processor according to the flowchart of FIG. 8 proceeds as follows when the processor addresses the entries # 11 , # 12 and # 13 in succession.
- the right to speak is on the partner side (steps 101 , 102 ) and the partner is supposed to utter a comical phrase.
- the processor proceeds to steps 103 , 104 to examine the audience field 34 , which contains the symbol “o”. Since the decision at step 104 is negative, flow proceeds to step 119 to make a search through the database 16 for a corresponding robot behavior by using the audience state currently analyzed by the analyzer 14 as a key.
- the processor performs the detected robot behavior. If the comedian partner utters some comments on the audience and their atmosphere, the robot will analyze the audience reaction to that comments and acts in accordance with the analyzed audience reaction.
- step 120 the right to speak is on the robot side (steps 101 , 102 ) and the robot proceeds to steps 115 , 116 to examine the audience held 34 , which contains the symbol “x”. Since the decision at step 116 is affirmative, flow proceeds to step 120 to read a comical phrase from the database 16 corresponding to the key B 006 specified in the robot behavior field 32 and utters the retrieved comical phrase.
- step 116 the right to speak is again on the robot side (steps 101 , 102 ) and the robot proceeds to steps 115 , 116 to examine the audience field 34 , which contains the symbol “o”. Since the decision at step 116 is negative, flow proceeds to step 119 to make a search through the database 16 for a corresponding robot behavior by using the audience state currently analyzed by the analyzer 14 as a key. At step 120 , the processor performs the detected robot behavior.
- FIG. 9 illustrates a modification of the first embodiment of this invention.
- the robot is additionally provided with a wireless interface 40 with which it establishes communication with a wireless LAN 41 .
- a wireless LAN 41 To the wireless LAN 41 is connected a server 42 , a microphone 43 and a CCD camera 44 .
- CCD camera 44 is located in such a position that it can take the whole view of the audience and the microphone 43 is located near the audience to pick up their voice.
- Server 42 is used to store data indicating the number of sold tickets and the social classes of the audience.
- the stored data is transmitted from the server 42 via the wireless LAN 41 to the robot where the wireless interface 40 receives the transmitted data for application to the audience state analyzer 14 .
- the outputs of the microphone 43 and camera 44 are also transmitted through the wireless LAN 41 to the robot as additional audience state data to the audience state analyzer 14 .
- mode A if (1) the partner has the right to speak, (2) no consideration of the audience is necessary and (3) the behavior of the partner is specified as A(x), then no behavior is performed by the robot if the recognized partner's behavior matches A(x). Otherwise, a specified robot behavior is performed.
- mode B if (1) the robot has the right to speak, (2) no consideration of the audience is necessary and (3) no behavior of the partner is specified, then a predetermined behavior R(x) is performed by the robot and the program proceeds to the next entry;
- mode C if (1) the robot has the right to speak, (2) consideration of the audience is necessary and (3) no behavior of the partner is specified, then a predetermined behavior R(x) is performed by the robot if the recognized state of the audience matches the expected state. Otherwise, the robot behavior database 16 is referenced using the recognized audience state;
- mode D if (1) the partner has the right to speak, (2) consideration of the audience is necessary and (3) the behavior of the partner is specified as A(x), then no behavior is performed by the robot if the recognized state of the audience matches the expected state. Otherwise, the robot behavior database 16 is searched with the recognized partner's behavior as a key to perform a corresponding robot behavior.
- the robot is used as assistance for a schoolteacher in a classroom for teaching a group of school children.
- the robot includes a teacher's behavior analyzer 52 to which the outputs of the microphone 9 , cameras 8 and touch sensor 10 and a teacher's feature memory 53 are supplied in a manner similar to the first embodiment.
- a pupils state analyzer 54 is provided to receive the outputs of the microphone 9 and cameras 8 to determine the reaction of the pupils in the classroom.
- scenario memory 55 and robot behavior database 56 are connected to the processor 57 .
- the robot is additionally provided with a teaching manual memory 58 for supplying the processor 57 with teaching materials including both textual and pictorial data.
- Scenario memory 55 includes a plurality of phrases of a scenario to be uttered by the teacher and the robot in a manner similar to the previous embodiment for teaching a particular subject and may further include the identity of a teaching material to be displayed on the video screen in the classroom.
- Processor 57 transmits a teaching material supplied from the memory 58 to a wireless interface 59 , which communicates with a wireless LAN 60 at intervals in response to a timing control signal, which may be stored in the scenario memory 55 .
- a video screen 61 is provided for displaying the teaching material supplied from the robot.
- a plurality of computer terminals 62 are placed respectively on the pupil's desks.
- the teaching materials from the robot may also be displayed on the individual computer terminals 62 .
- the pupils use the computer terminals 62 to input their answer in response to a teacher's question.
- Such input data are transmitted through the wireless LAN 61 to the robot, where the wireless interface 59 repeats it to the pupils state analyzer 54 .
- the robot stands in a position close to the schoolteacher. They perform a dialogue according to a scenario stored in the scenario memory 55 .
- the dialogue will proceed in a different way depending on the pupils' reaction to what is being taught on a subject or how the subject is explained by the teacher. In this manner, an element of entertainment can be incorporated into the classroom teaching at appropriate timing so that the pupils keep their attention for an extended period of time on a subject of the type likely to cause distraction.
Abstract
For a joint performance of a dialogue between a human partner and a robot, the robot analyzes phrase and action of the partner to detect a recognized behavior of the partner and analyzes a state of audience listening to utterances from the partner and the robot to detect a recognized state of the audience. A scenario describing the dialogue is stored in entries of a memory. The memory is successively referenced entry by entry and a check is made for a match between an utterance by the partner or the robot to a reaction from the audience. Responsive to a currently detected audience state, a corresponding robot behavior is determined. Preferably, possible partner's behaviors and expected audience states are mapped in a database to specified robot behaviors. The database is searched for a specified robot behavior corresponding to a currently sensed partner behavior or a currently sensed audience state.
Description
- 1. Field of the Invention
- The present invention relates generally to robots, and more specifically to a robot for participating in a joint performance with a human partner and a method of controlling a robot. The robot is useful for jointly performing a comic dialogue with a comedian partner in a theater and for assisting a schoolteacher in a classroom.
- 2. Description of the Related Art
- Recent advances in microprocessors and robotics have culminated in the development of two-footed robots known under the trade names of “Aibo” and “Asimo”. Another example of robot is a nursing robot which is described in Japanese Patent Publication 2000-353012 for nursing the aged or the handicapped. The robot has an eye whose viewing point is determined with respect to the position of a user. In response to a voice command from the user, the robot interprets the voice command at the determined viewing point and determines its direction of motion.
- Although the known robots are capable of recognizing human voice and of using light and sound to express their own feelings, their performance is limited to responding to only a simple phrase or action by a user. Therefore, their performance is a series of mutually unrelated behaviors.
- However, a need exists to provide a robot capable of participating in a joint performance of dialogue with a human partner.
- It is therefore an object of the present invention to provide a robot and a method of controlling a robot for participating in a joint performance with a human partner.
- According to a first aspect of the present invention, there is provided a robot comprising a first analyzer for analyzing phrase and action of a human partner to detect a recognized behavior of the partner, a second analyzer for analyzing a state of a plurality of people listening to utterances from the partner and the robot to detect a recognized state of the people, a scenario memory for storing a scenario describing a dialogue between the partner and the robot, and a processor fox making reference to a portion of the scenario in the memory according to one of the recognized behavior of the partner and the recognized state of the people, and determining a behavior of the robot according to the referenced portion of the scenario.
- According to a second aspect, the present invention provides a robot control system for controlling a robot, comprising a first analyzer for analyzing phrase and action of a human partner to detect a recognized behavior of the partner, a second analyzer for analyzing a state of a plurality of people listening to utterances from the partner and the robot to detect a recognized state of the people, a scenario memory for storing a scenario describing a dialogue between the partner and the robot, and a processor for making reference to a portion of the scenario in the memory according to one of the recognized behavior of the partner and the recognized state of the people, and determining a behavior of the robot according to the referenced portion of the scenario.
- According to a third aspect, the present invention provides a method of controlling a robot, comprising the steps of (a) storing a scenario describing a dialogue between a human partner and the robot in a memory, (b) analyzing phrase and action of a human partner to detect a recognized behavior of the partner and analyzing a state of a plurality of people listening to utterances from the partner and the robot to detect a recognized state of the people, and (c) making reference to a portion of the scenario in the memory according to at least one of the recognized behavior of the partner and the recognized state of the people and determining a behavior of the robot according to the referenced portion of the scenario.
- According to a fourth aspect, the present invention provides a computer-readable storage medium containing a stored scenario describing a dialogue between a human partner and a robot and a program for controlling the robot, the program performing the steps of (a) analyzing phrase and action of a human partner to detect a recognized behavior of the partner and analyzing a state of a plurality of people listening to utterances from the partner and the robot to detect a recognized state of the people, and (b) making reference to a portion of the stored scenario according to at least one of the recognized behavior of the partner and the recognized state of the people and determining a behavior of the robot according to the referenced portion of the scenario.
- The present invention will be described in detail further with reference to the following drawings, in which;
-
FIGS. 1A and 1B are front and side views of an entertainment robot of the present invention, respectively; -
FIG. 2 is a block diagram of the robot of according to a first embodiment of the present invention, in which the robot ads as an artificial partner for jointly performing a comic dialogue with a human comedian partner; -
FIG. 3 is an illustration of the scenario memory ofFIG. 2 according to the first embodiment of the invention; -
FIG. 4 is an illustration of the action pattern memory of the robot behavior database ofFIG. 2 for comical actions to be performed by the robot; -
FIG. 5 is an illustration of the phrase pattern memory of the robot behavior database for showing comical phrases to be uttered by the robot; -
FIG. 6 is a flowchart of the operation of the processor ofFIG. 2 according to the first embodiment; -
FIG. 7 is an illustration of a modified scenario memory according to a modification of the first embodiment of this invention; -
FIG. 8 is a flowchart of the operation of the processor according to the modification of the invention; -
FIG. 9 is a block diagram of the robot according to a farther modification of the first embodiment of the invention; and -
FIG. 10 is a block diagram of a robot according to a second embodiment of the present invention, in which the robot acts as an artificial assistance for a schoolteacher in a classroom for teaching a group of pupils. - According to a first embodiment of the present invention, an entertainment robot is shown in
FIGS. 1A and 1B . The entertainment robot is used on the stage of a hall or theater as an artificial comedian, standing on the side of a human comedian partner. They perform a comic dialogue by exchanging comical phrases with each other on a wide range of topics including social, political and human affairs. - The entertainment robot comprises a
torso 1 of generally cylindrical structure with a roundedupper portion 2 having acircular opening 3 and a pair oflimb portions 4 on its front Bide. Ahead 5 of spherical shape is partially submerged in theopening 3 and rotatably secured so that it can orient its face in a limited horizontal angle about avertical axis 5A and in a limited vertical angle about ahorizontal axis 5B. Each of thelimb portions 4 extends outwards from thetorso 1 and includes a drivenwheel 6. Acastor 7 is fitted to the bottom oftorso 1 near its rear side so that the robot is capable of moving about on the stage. -
Head 5 is provided with aCCD camera 8A in the position of an eye, athermographic sensor 8B in the position of the other eye and atouch sensor 10 on top of the head. Amicrophone 9 and aloudspeaker 11 are provided on the front side oftorso 1. - As shown in
FIG. 2 , the outputs ofCCD camera 8A,thermographic sensor 8B, microphone 9 andtouch sensor 10 are connected to a partner'sbehavior analyzer 12. Partner'sbehavior analyzer 12 uses its input signals to determine what the partner has behaved by consulting a partner's featurememory 13 in which a plurality of facial features and speech patterns of the partner are stored as references. Analyzer 12 uses thethermographic sensor 8B to analyze the feeling of the partner to detect a recognized partner's behavior. - The outputs of
CCD camera 8A andmicrophone 9 are coupled to anaudience state analyzer 14, which determines whether the theater audience is large or small and analyzes their reaction in response to amusing phrases or serious remarks from the performers. - A
scenario memory 15 and arobot behavior database 16 are provided.Robot behavior database 16 includes anaction pattern memory 16A and aphrase pattern memory 16B. The details of these memories will be described below. - The robot includes a pair of
head actuators head 5 in vertical and horizontal angular positions and a pair ofwheel actuators Controller 20 responds to the output signal from theprocessor 17 to individually control these actuators. - As shown in
FIG. 3 , thescenario memory 15 is divided into a plurality of entries (rows) numbered and arranged according to the scenario of a comic dialogue being played by the comedian partner and the robot. Each entry is sub-divided into a plurality of fields (columns) 31 to 34. - A specified partner's action or phrase is indicated in the
field 31, and one or more database keys (A001 and B001, for example) are indicated in thefield 32 for making a search through therobot database 16 for corresponding robot action and phrase. Thefield 33 of each entry is a right-to-speak field that indicates which of the comedian or robot has the right to speak first in that entry. Thefield 34 is an audience field for which the robot is programmed to consider an expected scale of audience (large or small) and an expected reaction of audience (laughter or no laughter). The symbol “x” in theaudience field 34 indicates that there is no need for the robot to consider audience reaction. -
Robot behavior field 32 of entries numbered 3,5,6, and 10 is further sub-divided into on upper sub-field and a lower sub-field. In the case ofentry # 3, the lower sub-field specifies an alternative robot's behavior to be performed when the partner fails to do according to the scenario specified in the partner'sbehavior field 31. In theentries # 5, #6 and #10, the lower sub-field specifies an alternative robot's behavior to be performed when sensed audience state (scale or reaction) does not match the expected audience state specified in theaudience field 34. - As shown in
FIG. 4 , theaction pattern memory 16A ofrobot behavior database 16 has a plurality of entries (rows) in which database keys A001 through A008 are mapped to corresponding patterns of robot action and sensed partner's behaviors, which will be detected by theanalyzer 12. Using the database key and a sensed partner's behavior, theaction pattern memory 16A is searched for a corresponding pattern of action, which the robot is programmed to perform. The robot actions may include bowing, turning around, stepping forward and combinations of those actions. - As shown in
FIG. 5 , thephrase pattern memory 16B ofrobot behavior database 16 has a plurality of entries (rows) in which database keys B001 through B008 are mapped to corresponding phrase patterns of the robot and sensed partner's behaviors, which will be detected by theanalyzer 12. Using the database key and a sensed partner's behavior, thephrase pattern memory 16B is searched for a corresponding phrase pattern, which the robot is programmed to utter. - In the above example,
FIG. 3 shows phrase keys B001 to B007 ofFIG. 4 . In the entries of thescenario memory 15, action keys specified inFIG. 5 may be combined with the phrase keys to indicate a desired robot behavior. - The operation of
processor 17 proceeds according to the flowchart ofFIG. 6 . -
Processor 17 initially sets a pointer of thescenario memory 15 to the first entry and begins withstep 101 by examining the right-to-speakfield 33 of the entry of thescenario memory 15 indicated by the pointer. Atstep 102, theprocessor 17 determines which side has the right to speak, the comedian partner ox the robot. If the partner has the right to speak, flow proceeds fromstep 102 to step 103 to examine theaudience field 34 of the current entry and flow proceeds to step 104 to determine whether it is necessary to consider the audience reaction or state. If theaudience field 34 is marked with the symbol “x”, it is not necessary to consider the audience state and flow proceeds to step 105 to compare a partner's behavior (uttered phrase) detected by theanalyzer 12 to the phrase specified in the partner'sbehavior field 31 of the current entry to detect a match (step 106). If they match, flow proceeds fromstep 106 to step 121 to check to see if the current entry is the last entry. If not, flow proceeds to step 122 to advance the pointer to the next entry. If the decision atstep 106 is negative, flow proceeds to step 107 to read a robot behavior data from the lower sub-field of robot'sbehavior field 32. - If the partner fails to behave according to the scenario specified in the partner's
behavior field 31 of the current entry, the decision atstep 106 is negative and robot behavior data is read from the lower sub-field ofrobot behavior field 32 for urging the partner to perform the specified behavior. If the partner has failed to perform the specified behavior a predetermined number of times (step 108), flow proceeds to step 121. Otherwise, flow proceeds to step 109 to perform the specified behavior, and flow returns to step 105 to repeat the process. - If the
audience field 34, examined atstep 103, contains an expected audience state, the decision atstep 104 is negative and flow proceeds to step 111 to turn the robot face towards the audience to sense the audience state by theaudience state analyzer 14 and compares the sensed audience to the expected audience state for a match (step 112). If they match, flow proceeds to step 121. Otherwise, flow proceeds to step 113 to turn the robot face toward the partner to sense his behavior by the partner'sbehavior analyzer 12 and makes a search through therobot behavior database 16 for a corresponding robot behavior to be performed, using the sensed partner's behavior as a key. If a corresponding robot behavior is specified in the database 16 (step 114), flow proceeds to step 120 to perform a robot behavior specified in the robot behavior field 32 (or robot behavior database 16). Otherwise, flow proceeds to step 121. - If the right-to-speak
field 33, referenced atstep 101, indicates that the robot has the right to speak first in the current entry, flow proceeds fromstep 102 to step 115 to examine theaudience field 34. If theaudience field 34 contains an expected audience state, the decision atstep 116 is negative and flow proceeds to step 117 to turn the robot face toward the audience to detect the audience state by theanalyzer 14 and compares the sensed audience state to the expected state for a match (step 118). If they match, flow proceeds to step 120. Otherwise, flow proceeds to step 119 to read a key from the lower sub-field of therobot behavior field 32. Flow proceeds to step 120 to perform the robot behavior specified in thedatabase 16 corresponding to the retrieved key. - The following is a description of the operation of the
processor 17 according to the contents ofscenario memory 15 with reference toFIGS. 3 , 4 and 5. - First, the processor initializes the address pointer to “1” and examines the right-to-speak
field 33 ofentry # 1 of the scenario memory 15 (step 101) and proceeds to step 102. In theentry # 1, the right to speak is on the partner side and the robot's human partner is supposed to utter a self-introduction phrase as indicated in the partner'sbehavior field 31 and the processor proceeds tosteps audience field 34. Since the audience field ofentry # 1 is marked with the symbol “x”, flow proceeds to step 105 to turn the robot face toward the partner to sense his behavior (i.e., the uttered self-introduction phrase) by theanalyzer 12 and compares the partner's behavior to the phrase specified in the partner'sbehavior field 31 for a match. If they match, flow proceeds throughsteps entry # 2 of the scenario memory. Therefore, the processor performs the scenario ofentry # 1 by following steps 101-106, 121 and 122 (as indicated aspass 1 inFIG. 3 ). - In the
entry # 2, the right to speak is on the robot aide (steps 101, 102). The processor proceeds fromstep 102 to step 115 to examine theaudience field 34. Since it is marked with the symbol “yd”, the decision atstep 116 is affirmative and the processor proceeds to step 120 to perform the robot behavior specified in therobot behavior field 32. Since the key B001 is marked in therobot behavior field 32, the robot is instructed to utter a self-introduction phrase such as “My name is PAPERO”, and than proceeds throughsteps entry # 2 by followingsteps pass 2 inFIG. 3 ). - In the
entry # 3, the right to speak is on the partner side (steps 101, 102) and the partner is supposed to perform the behavior specified in thefield 31. In this case, the partner is supposed to slap the robot on the head (i.e., the touch sensor 10). The processor proceeds tosteps audience field 34. Since the audience field ofentry # 3 is marked with the symbol “x”, flow proceeds to step 105 to compare the partner's behavior (i.e., the slapping on the robot's head) to the action specified in the partner'sbehavior field 31 for a match (step 106). If they match, flow proceeds fromstep 106 to step 122 viastep 121 to increment the address pointer by one and returns to step 101. Therefore, the processor follows thepass 1 as in the case ofentry # 1 if the partner's behavior matches the specified scenario. If the partner falls to make a slapping action, the decision atstep 106 is negative and the processor reads an urging phrase (“Slap My Head”) from thedatabase 16 corresponding to the key B002 specified in the robot's behavior field 32 (step 107) and flow proceeds throughstep 108 to step 109 to utter the retrieved urging phrase, and flow returns to step 105. As a result, when the partner fails to behave according to the specified scenario the processor performs steps 101-109, 121 and 122 (pass 3). - In the
entry # 4, the right to speak is again on the partner side (steps 101, 102) and the partner is supposed to utter a greeting phrase such as “Happy to See All of You!” The processor proceeds tosteps audience field 34. Audience field ofentry # 4 is again marked with the symbol “x”. Thus, flow proceeds to step 105 to compare the partner's utterance (i.e., the greeting phrase) to the phrase specified in the partner'sbehavior field 31 for a match (step 106). If they match, flow proceeds fromstep 106 to step 122 viastep 121 to increment the address pointer to “5” and returns to step 101. If the partner forgets to utter the greeting phrase, the processor performs steps 101-109, 121 and 122 (pass 3) to read an urging phrase from thedatabase 16 corresponding to the key B003 specified in the robot'sbehavior field 32 ofentry # 3 to urge the robot's partner to utter the missing phrase. - In the
entry # 5, the right to speak is on the robot side (steps 101, 102). The processor proceeds fromstep 102 to step 115 to examine theaudience field 34, which contains an expected state of audience (large audience). The decision atstep 116 is therefore negative and flow proceeds to step 117 to compare the sensed audience size (i.e., the number of people) to the expected large audience for a match (step 118). If they match, flow proceeds to step 120 to read a reactive phrase (“Hello, Everybody”) from thedatabase 16 corresponding to the key B004 specified in the upper sub-field ofrobot behavior field 32 and utter the retrieved phrase. Atstep 122, the processor increments the address pointer to “6” before returning to step 101. In this example, steps 101, 102, 115-118, 120-122 (pass 4) are executed. - If the sensed audience is small, the decision at
step 118 is negative and flow proceeds to step 119 to make a search through thedatabase 16 for one of the reactive behaviors using the output of theaudience state analyzer 14 and performs the detected reactive behavior (step 120). In this case, steps 101, 102, 115-122 (pass 5) are executed. - In the
entry # 6, the right to speak is on the partner side (steps 101, 102) and the partner is supposed to utter an opening phrase such as “Today, We Are Going to Perform a Comic Dialogue!” The processor proceeds tosteps audience field 34, which indicates large audience. Since the decision atstep 104 is negative, flow proceeds to step 111 to compare the sensed audience state to the expected large audience for a match (step 112). If they match, flow proceeds throughstep 121 to step 122 to increment the address counter to “7”. If the comparison atstep 111 indicates that the sensed audience is small, the decision atstep 112 is negative and flow proceeds to step 113 to make a search through thedatabase 16 for a reactive robot behavior using the partner's behavior currently sensed by the partner'sbehavior analyzer 12 as a key. If a corresponding reactive robot behavior is detected (step 114), flow proceeds to step 120 to perform the detected behavior. - In the
entry # 7, the right to speak is on the robot side (steps 101, 102). The processor proceeds fromstep 102 to step 115 to examine theaudience field 34, which contains an “X” mark. The decision atstep 116 is therefore affirmative and flow proceeds to step 120 to read a comical phrase from thedatabase 16 corresponding to the key B005 specified in therobot behavior field 32 and utter the retrieved comical phrase. Atstep 122, the processor increments the address pointer to “8” before returning to step 101. In this example, steps 101, 102, 115, 116, 120-122 (pass 2) are executed. - In the
entry # 8, the right to speak is again on the robot side (steps 101, 102). The processor proceeds fromstep 102 to step 115 to examine theaudience field 34, which contains a “laughter” mark as an expected audience reaction. The decision atstep 116 is negative and flow proceeds to step 117 to compare the output of theaudience state analyzer 14 to the expected audience reaction for a match (step 118). If the robot's comical phrase ofentry # 7 has appealed to the audience, causing them to laugh, the decision atstep 118 is affirmative, and flow proceeds to step 120 to read a reactive phrase from thedatabase 16 corresponding to the key B006 specified in therobot behavior field 32 and utters the retrieved comical phrase. In this example, steps 101, 102, 115-118, 120-122 (pass 4) are executed, incrementing the address pointer to “9”. - If the robot's comical phrase of
entry # 7 has not appealed to the audience, causing them to remain silent, the decision atstep 118 for thecurrent entry # 8 is negative, and flow proceeds to step 119 to read reactive motion data from thedatabase 16 corresponding to the audience reaction sensed by theaudience state analyzer 14 and performs the retrieved reactive motion (step 120). In this example, steps 101, 102, 115-122 (pass 5) are executed. - In the
entry # 9, the right to speak is on the partner side (steps 101, 102) and the partner is supposed to utter a comical phrase. The processor proceeds tosteps audience field 34, which indicates a laughter mark. Since the decision atstep 104 is negative, flow proceeds to step 111 to compare the sensed audience state to the expected laughing state of the audience for a match (step 112). If the partner's comical phrase during the current entry is appealing to the audience, the decision atstep 112 is affirmative and flow proceeds throughstep 121 to step 122 to increment the address pointer to the next. If the current partner's comical phrase is not appealing to the audience, the decision atstep 112 is negative and flow proceeds to step 113 to make a search through thedatabase 16 for a reactive robot behavior using the partner's behavior currently sensed by the partner'sbehavior analyzer 12 as a key. If a corresponding reactive robot behavior is detected (step 114), flow proceeds to step 120 to perform the detected behavior. -
Scenario memory 15 is modified to includefurther entries # 11, #12, and #13 as shown inFIG. 7 and the flowchart ofFIG. 6 is modified as shown inFIG. 8 . - In
FIG. 7 , the audience held 34 of each entry further includes a symbol “o” which indicates that theprocessor 17 should recognize the analyzed audience state. Depending on the recognized state, the processor determines its reactive phrase or motion. - The operation of the processor according to the flowchart of
FIG. 8 proceeds as follows when the processor addresses theentries # 11, #12 and #13 in succession. - In the
entry # 11, the right to speak is on the partner side (steps 101, 102) and the partner is supposed to utter a comical phrase. The processor proceeds tosteps audience field 34, which contains the symbol “o”. Since the decision atstep 104 is negative, flow proceeds to step 119 to make a search through thedatabase 16 for a corresponding robot behavior by using the audience state currently analyzed by theanalyzer 14 as a key. Atstep 120, the processor performs the detected robot behavior. If the comedian partner utters some comments on the audience and their atmosphere, the robot will analyze the audience reaction to that comments and acts in accordance with the analyzed audience reaction. - In the
entry # 12, the right to speak is on the robot side (steps 101, 102) and the robot proceeds tosteps step 116 is affirmative, flow proceeds to step 120 to read a comical phrase from thedatabase 16 corresponding to the key B006 specified in therobot behavior field 32 and utters the retrieved comical phrase. - In the
entry # 13, the right to speak is again on the robot side (steps 101, 102) and the robot proceeds tosteps audience field 34, which contains the symbol “o”. Since the decision atstep 116 is negative, flow proceeds to step 119 to make a search through thedatabase 16 for a corresponding robot behavior by using the audience state currently analyzed by theanalyzer 14 as a key. Atstep 120, the processor performs the detected robot behavior. - In this way, enhanced reality can be given to the stage performance of a comic dialogue by a comedian and a robot by adaptively controlling the robot to different audience reactions.
-
FIG. 9 illustrates a modification of the first embodiment of this invention. In this modification, the robot is additionally provided with awireless interface 40 with which it establishes communication with awireless LAN 41. To thewireless LAN 41 is connected aserver 42, amicrophone 43 and aCCD camera 44.CCD camera 44 is located in such a position that it can take the whole view of the audience and themicrophone 43 is located near the audience to pick up their voice.Server 42 is used to store data indicating the number of sold tickets and the social classes of the audience. The stored data is transmitted from theserver 42 via thewireless LAN 41 to the robot where thewireless interface 40 receives the transmitted data for application to theaudience state analyzer 14. The outputs of themicrophone 43 andcamera 44 are also transmitted through thewireless LAN 41 to the robot as additional audience state data to theaudience state analyzer 14. - It is seen from the foregoing description, the operation of the robot can be summarized as the following six modes:
- In mode A, if (1) the partner has the right to speak, (2) no consideration of the audience is necessary and (3) the behavior of the partner is specified as A(x), then no behavior is performed by the robot if the recognized partner's behavior matches A(x). Otherwise, a specified robot behavior is performed.
- In mode B, if (1) the robot has the right to speak, (2) no consideration of the audience is necessary and (3) no behavior of the partner is specified, then a predetermined behavior R(x) is performed by the robot and the program proceeds to the next entry;
- In mode C, if (1) the robot has the right to speak, (2) consideration of the audience is necessary and (3) no behavior of the partner is specified, then a predetermined behavior R(x) is performed by the robot if the recognized state of the audience matches the expected state. Otherwise, the
robot behavior database 16 is referenced using the recognized audience state; - In mode D, if (1) the partner has the right to speak, (2) consideration of the audience is necessary and (3) the behavior of the partner is specified as A(x), then no behavior is performed by the robot if the recognized state of the audience matches the expected state. Otherwise, the
robot behavior database 16 is searched with the recognized partner's behavior as a key to perform a corresponding robot behavior. - In a second embodiment of the present invention, the robot is used as assistance for a schoolteacher in a classroom for teaching a group of school children. As shown in
FIG. 10 , the robot includes a teacher'sbehavior analyzer 52 to which the outputs of themicrophone 9,cameras 8 andtouch sensor 10 and a teacher'sfeature memory 53 are supplied in a manner similar to the first embodiment. Apupils state analyzer 54 is provided to receive the outputs of themicrophone 9 andcameras 8 to determine the reaction of the pupils in the classroom. Similar to the first embodiment,scenario memory 55 androbot behavior database 56 are connected to theprocessor 57. - In this embodiment, the robot is additionally provided with a teaching
manual memory 58 for supplying theprocessor 57 with teaching materials including both textual and pictorial data.Scenario memory 55 includes a plurality of phrases of a scenario to be uttered by the teacher and the robot in a manner similar to the previous embodiment for teaching a particular subject and may further include the identity of a teaching material to be displayed on the video screen in the classroom.Processor 57 transmits a teaching material supplied from thememory 58 to awireless interface 59, which communicates with awireless LAN 60 at intervals in response to a timing control signal, which may be stored in thescenario memory 55. - In the classroom, a
video screen 61 is provided for displaying the teaching material supplied from the robot. Preferably, a plurality ofcomputer terminals 62 are placed respectively on the pupil's desks. The teaching materials from the robot may also be displayed on theindividual computer terminals 62. The pupils use thecomputer terminals 62 to input their answer in response to a teacher's question. Such input data are transmitted through thewireless LAN 61 to the robot, where thewireless interface 59 repeats it to the pupils stateanalyzer 54. - In the classroom, the robot stands in a position close to the schoolteacher. They perform a dialogue according to a scenario stored in the
scenario memory 55. The dialogue will proceed in a different way depending on the pupils' reaction to what is being taught on a subject or how the subject is explained by the teacher. In this manner, an element of entertainment can be incorporated into the classroom teaching at appropriate timing so that the pupils keep their attention for an extended period of time on a subject of the type likely to cause distraction.
Claims (17)
1-13. (canceled)
14. A robot control system for controlling a robot, comprising:
a first analyzer for analyzing phrase and action of a human partner to detect a recognized behavior of the partner;
a second analyzer for analyzing a state of a plurality of people listening to utterances from said partner and said robot to detect a recognized state of said people;
a scenario memory for storing a scenario describing a dialogue between said partner and said robot; and
a processor for making reference to a portion of said scenario in said memory according to the recognized behavior of said partner and the recognized state of said people, and determining a behavior of said robot according to the referenced portion of said scenario.
15. The robot control system of claim 14 , wherein said scenario memory includes a plurality of entries, each of said entries containing a portion of said scenario to be uttered by one of said partner and said robot,
wherein said processor successively makes reference to each of said entries, produces a predetermined utterance from the robot according to the referenced portion of said scenario and determines said behavior of said robot according to one of a recognized behavior of said partner currently detected by the first analyzer and a recognized state of said people currently detected by the second analyzer.
16. The robot control system of claim 15 , further comprising a robot behavior database for mapping a plurality of recognized behaviors of said partner which will be detected by said first analyzer to a plurality of predetermined robot behaviors,
wherein the referenced entry indicates an expected state of said people, and
wherein said processor compares a recognized state of said people currently detected by said second analyzer to the expected state of the referenced entry for a match or mismatch and determines said behavior of the robot according to the referenced portion of said scenario if there is a match between said recognized and expected states and according to one of said predetermined robot behaviors of said database corresponding to a recognized behavior of said partner currently detected by said first analyzer if there is a mismatch between said recognized and expected states.
17. The robot control of claim 15 , wherein each of said entries of said scenario memory indicates which one of said partner and said robot has the right to speak, and
wherein said processor determines said behavior of said robot according to a recognized behavior of said partner currently detected by said first analyzer if the referenced entry indicates that said partner has the right to speak and according to a state of said people currently recognized by said second analyzer if said referenced entry indicates that said robot has the right to speak.
18. The robot control system of claim 17 , wherein said processor compares a recognized behavior of said partner currently detected by the first analyzer to a predetermined behavior if the referenced entry indicates that said partner has the right to speak, and produces an utterance from the robot for urging the partner to perform said predetermined behavior if there is a mismatch between said recognized behavior said predetermined behavior.
19. The robot control system of claim 16 , wherein each of said entries of said scenario memory indicates which one of said partner and said robot has the right to speak,
wherein said database further maps a plurality of recognized states of said people which will be detected by said second analyzer to said plurality of predetermined robot behaviors,
wherein said processor determines said behavior of said robot according to one of said predetermined robot behaviors corresponding to a recognized behavior of said partner currently detected by said first analyzer if the referenced entry indicates that said partner has the right to speak and according to one of said predetermined robot behaviors corresponding to a recognized state of said people currently detected by said second analyzer if the referenced entry indicates that said robot has the right to speak.
20. The robot control system of claim 19 , wherein said processor compares a recognized behavior of said partner currently detected by the first analyzer to a predetermined partner's behavior if the referenced entry indicates that said partner has the right to speak, and produces an utterance from the robot for urging the partner to perform said predetermined partner's behavior if there is a mismatch between said recognized behavior said predetermined partner's behavior.
21. The robot control system of claim 15 , further comprising a robot behavior database for mapping a plurality of recognized states of said people which will be detected by said second analyzer to a plurality of predetermined robot behaviors, and
wherein said processor determines said behavior of the robot according to one of said predetermined robot behaviors of said database corresponding to a recognized state of said people currently detected by said second analyzer.
22. The robot control system of claim 14 , wherein said dialogue stored in said scenario memory is a comic dialogue.
23. The robot control system of claim 14 , wherein said first analyzer recognizes the behavior of said partner from a microphone and a video camera.
24. The robot control system of claim 14 , wherein said second analyzer recognizes the state of said people from a microphone and a video camera.
25. The robot control system of claim 14 , wherein said first analyzer recognizes the state of said partner from a thermographic sensor and a touch sensor for producing a signal when the touch sensor is in contact with said partner.
26. The robot control system of claim 24 , wherein said robot has a torso and a head rotatable on said torso about a vertical axis, and wherein said video camera is mounted on said head.
27. (canceled)
28. The robot control system of claim 14 , further comprising:
a video screen located external to said robot;
a plurality of computer terminals located external to said robot;
a network for establishing a wireless link from said processor to said video screen and a wireless link from said computer terminals to said processor; and
a teaching manual memory for storing teaching materials,
wherein said processor reads a teaching material from said teaching manual memory and transmitting the material to said video screen and collecting data from said computer terminals.
29-46. (canceled)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/401,873 US20090177321A1 (en) | 2004-04-28 | 2009-03-11 | Robot for participating in a joint performance with a human partner |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2004132789 | 2004-04-28 | ||
JP2004-132789 | 2004-04-28 | ||
JP2004241523A JP4622384B2 (en) | 2004-04-28 | 2004-08-20 | ROBOT, ROBOT CONTROL DEVICE, ROBOT CONTROL METHOD, AND ROBOT CONTROL PROGRAM |
JP2004-241523 | 2004-08-20 | ||
US11/116,436 US7526363B2 (en) | 2004-04-28 | 2005-04-28 | Robot for participating in a joint performance with a human partner |
US12/401,873 US20090177321A1 (en) | 2004-04-28 | 2009-03-11 | Robot for participating in a joint performance with a human partner |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/116,436 Division US7526363B2 (en) | 2004-04-28 | 2005-04-28 | Robot for participating in a joint performance with a human partner |
Publications (1)
Publication Number | Publication Date |
---|---|
US20090177321A1 true US20090177321A1 (en) | 2009-07-09 |
Family
ID=35188137
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/116,436 Active 2027-05-14 US7526363B2 (en) | 2004-04-28 | 2005-04-28 | Robot for participating in a joint performance with a human partner |
US12/401,927 Abandoned US20090182453A1 (en) | 2004-04-28 | 2009-03-11 | Robot for participating in a joint performance with a human partner |
US12/401,873 Abandoned US20090177321A1 (en) | 2004-04-28 | 2009-03-11 | Robot for participating in a joint performance with a human partner |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/116,436 Active 2027-05-14 US7526363B2 (en) | 2004-04-28 | 2005-04-28 | Robot for participating in a joint performance with a human partner |
US12/401,927 Abandoned US20090182453A1 (en) | 2004-04-28 | 2009-03-11 | Robot for participating in a joint performance with a human partner |
Country Status (2)
Country | Link |
---|---|
US (3) | US7526363B2 (en) |
JP (1) | JP4622384B2 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105892469A (en) * | 2016-06-14 | 2016-08-24 | 苏州迈奇杰智能技术有限公司 | Multi-memory based visible light control management system for robot |
CN106325118A (en) * | 2015-06-30 | 2017-01-11 | 芋头科技(杭州)有限公司 | Robot active degree intelligent control system and method |
CN112147929A (en) * | 2020-09-16 | 2020-12-29 | 芜湖宏嘉科技有限公司 | Intelligent classroom system for teaching |
Families Citing this family (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8924021B2 (en) * | 2006-04-27 | 2014-12-30 | Honda Motor Co., Ltd. | Control of robots from human motion descriptors |
WO2008076445A2 (en) * | 2006-12-18 | 2008-06-26 | Disney Enterprises, Inc. | Method, system and computer program product for providing group interactivity with entertainment experiences |
KR100866212B1 (en) * | 2007-02-08 | 2008-10-30 | 삼성전자주식회사 | Genetic robot platform and genetic robot behavior manifestation method |
US8060255B2 (en) * | 2007-09-12 | 2011-11-15 | Disney Enterprises, Inc. | System and method of distributed control of an interactive animatronic show |
US8140188B2 (en) * | 2008-02-18 | 2012-03-20 | Toyota Motor Engineering & Manufacturing North America, Inc. | Robotic system and method for observing, learning, and supporting human activities |
JP2009262279A (en) * | 2008-04-25 | 2009-11-12 | Nec Corp | Robot, robot program sharing system, robot program sharing method, and program |
TWI392983B (en) * | 2008-10-06 | 2013-04-11 | Sonix Technology Co Ltd | Robot apparatus control system using a tone and robot apparatus |
CN101727074B (en) * | 2008-10-24 | 2011-12-21 | 鸿富锦精密工业(深圳)有限公司 | Biology-like device with biological clock and behavior control method thereof |
FR2965375B1 (en) * | 2010-09-27 | 2013-04-05 | Ivan Lovric | SYSTEM AND METHOD FOR DIGITAL CONVERSATIONAL AGENT WITH EVOLVING CHARACTER |
CN104914815A (en) * | 2015-04-15 | 2015-09-16 | 北汽福田汽车股份有限公司 | Processor monitoring method, device and system |
WO2018135276A1 (en) * | 2017-01-19 | 2018-07-26 | シャープ株式会社 | Speech and behavior control device, robot, control program, and control method for speech and behavior control device |
JP6822244B2 (en) * | 2017-03-17 | 2021-01-27 | ヤマハ株式会社 | Control method and control device |
JP2019053351A (en) * | 2017-09-12 | 2019-04-04 | 大日本印刷株式会社 | Reception apparatus, computer program and reception method |
JP6768613B2 (en) * | 2017-09-14 | 2020-10-14 | 日本電信電話株式会社 | Speech processing equipment, methods and programs |
CN109015664A (en) * | 2018-05-31 | 2018-12-18 | 芜湖星途机器人科技有限公司 | Control system for robot |
JP2018205771A (en) * | 2018-09-05 | 2018-12-27 | カシオ計算機株式会社 | Robot control device, robot control method, and program |
KR20200079054A (en) * | 2018-12-24 | 2020-07-02 | 엘지전자 주식회사 | Robot and method for controlling thereof |
CN113942009B (en) * | 2021-09-13 | 2023-04-18 | 苏州大学 | Robot bionic hand grabbing method |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US210345A (en) * | 1878-11-26 | Improvement in removable covers for show-boxes | ||
US4221927A (en) * | 1978-08-08 | 1980-09-09 | Scott Dankman | Voice responsive "talking" toy |
US5281143A (en) * | 1992-05-08 | 1994-01-25 | Toy Biz, Inc. | Learning doll |
US6292713B1 (en) * | 1999-05-20 | 2001-09-18 | Compaq Computer Corporation | Robotic telepresence system |
US6585556B2 (en) * | 2000-05-13 | 2003-07-01 | Alexander V Smirnov | Talking toy |
US6971943B1 (en) * | 2003-09-30 | 2005-12-06 | Arne Schulze | Interactive sound producing toy |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH09131468A (en) * | 1995-11-09 | 1997-05-20 | Matsushita Electric Ind Co Ltd | Dolls of a pair of comic dialogists |
JP2000155606A (en) * | 1998-11-24 | 2000-06-06 | Ricoh Elemex Corp | Operation control system |
JP3318539B2 (en) | 1999-06-10 | 2002-08-26 | 株式会社エイ・ティ・アール知能映像通信研究所 | Autonomous mobile robot with dialogue system |
JP2001273473A (en) * | 2000-03-24 | 2001-10-05 | Atr Media Integration & Communications Res Lab | Agent for conversation and conversation system using the same |
JP2004237392A (en) * | 2003-02-05 | 2004-08-26 | Sony Corp | Robotic device and expression method of robotic device |
JP2004024867A (en) * | 2003-06-09 | 2004-01-29 | Matsushita Electric Ind Co Ltd | Voice interaction toy |
US20050154265A1 (en) * | 2004-01-12 | 2005-07-14 | Miro Xavier A. | Intelligent nurse robot |
-
2004
- 2004-08-20 JP JP2004241523A patent/JP4622384B2/en not_active Expired - Fee Related
-
2005
- 2005-04-28 US US11/116,436 patent/US7526363B2/en active Active
-
2009
- 2009-03-11 US US12/401,927 patent/US20090182453A1/en not_active Abandoned
- 2009-03-11 US US12/401,873 patent/US20090177321A1/en not_active Abandoned
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US210345A (en) * | 1878-11-26 | Improvement in removable covers for show-boxes | ||
US4221927A (en) * | 1978-08-08 | 1980-09-09 | Scott Dankman | Voice responsive "talking" toy |
US5281143A (en) * | 1992-05-08 | 1994-01-25 | Toy Biz, Inc. | Learning doll |
US6292713B1 (en) * | 1999-05-20 | 2001-09-18 | Compaq Computer Corporation | Robotic telepresence system |
US6585556B2 (en) * | 2000-05-13 | 2003-07-01 | Alexander V Smirnov | Talking toy |
US6971943B1 (en) * | 2003-09-30 | 2005-12-06 | Arne Schulze | Interactive sound producing toy |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106325118A (en) * | 2015-06-30 | 2017-01-11 | 芋头科技(杭州)有限公司 | Robot active degree intelligent control system and method |
CN105892469A (en) * | 2016-06-14 | 2016-08-24 | 苏州迈奇杰智能技术有限公司 | Multi-memory based visible light control management system for robot |
CN112147929A (en) * | 2020-09-16 | 2020-12-29 | 芜湖宏嘉科技有限公司 | Intelligent classroom system for teaching |
Also Published As
Publication number | Publication date |
---|---|
US7526363B2 (en) | 2009-04-28 |
JP2005335053A (en) | 2005-12-08 |
US20090182453A1 (en) | 2009-07-16 |
JP4622384B2 (en) | 2011-02-02 |
US20050246063A1 (en) | 2005-11-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7526363B2 (en) | Robot for participating in a joint performance with a human partner | |
CN108000526B (en) | Dialogue interaction method and system for intelligent robot | |
JP6816925B2 (en) | Data processing method and equipment for childcare robots | |
EP1482480B1 (en) | Learning apparatus, learning method, and robot apparatus | |
US7987091B2 (en) | Dialog control device and method, and robot device | |
US20030220796A1 (en) | Dialogue control system, dialogue control method and robotic device | |
US20180257236A1 (en) | Apparatus, robot, method and recording medium having program recorded thereon | |
WO2002091356A1 (en) | Obot device, character recognizing apparatus and character reading method, and control program and recording medium | |
JP2002351305A (en) | Robot for language training | |
Tsiami et al. | Multi3: Multi-sensory perception system for multi-modal child interaction with multiple robots | |
JP6308608B2 (en) | Foreign language conversation learning system based on direct method | |
JP4062591B2 (en) | Dialog processing apparatus and method, and robot apparatus | |
JP7096626B2 (en) | Information extraction device | |
KR20010007842A (en) | The system and method of a dialogue form voice and multi-sense recognition for a toy | |
CN106251717A (en) | Intelligent robot speech follow read learning method and device | |
JP2022075661A (en) | Information extraction apparatus | |
JP6972526B2 (en) | Content providing device, content providing method, and program | |
KR102426792B1 (en) | Method for recognition of silent speech and apparatus thereof | |
KR20060091329A (en) | Interactive system and method for controlling an interactive system | |
US20210201139A1 (en) | Device and method for measuring a characteristic of an interaction between a user and an interaction device | |
WO2022091230A1 (en) | Information extraction device | |
JP3919726B2 (en) | Learning apparatus and method | |
JP2001209779A (en) | Virtual biological system and pattern learning method in virtual biological system | |
Roy | Grounded speech communication | |
JP3363229B2 (en) | Character learning device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |