WO2011067469A1 - A method, a system and a computer program for instructing and evaluating full-body movement - Google Patents

A method, a system and a computer program for instructing and evaluating full-body movement Download PDF

Info

Publication number
WO2011067469A1
WO2011067469A1 PCT/FI2010/050990 FI2010050990W WO2011067469A1 WO 2011067469 A1 WO2011067469 A1 WO 2011067469A1 FI 2010050990 W FI2010050990 W FI 2010050990W WO 2011067469 A1 WO2011067469 A1 WO 2011067469A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
instruction
scrolling
instructions
preview
Prior art date
Application number
PCT/FI2010/050990
Other languages
French (fr)
Inventor
Perttu HÄMÄLÄINEN
Original Assignee
Virtual Air Guitar Company Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Virtual Air Guitar Company Oy filed Critical Virtual Air Guitar Company Oy
Publication of WO2011067469A1 publication Critical patent/WO2011067469A1/en

Links

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B24/00—Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
    • A63B24/0003—Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
    • A63B24/0006—Computerised comparison for qualitative assessment of motion sequences or the course of a movement
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/80—Special adaptations for executing a specific game genre or game mode
    • A63F13/833—Hand-to-hand fighting, e.g. martial arts competition
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20—Input arrangements for video game devices
    • A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/213—Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/40—Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
    • A63F13/44—Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment involving timing of operations, e.g. performing an action within a time slot
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1087—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera
    • A63F2300/1093—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera using visible light
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/60—Methods for processing data by generating or executing the game program
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/60—Methods for processing data by generating or executing the game program
    • A63F2300/6009—Methods for processing data by generating or executing the game program for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
    • A63F2300/6018—Methods for processing data by generating or executing the game program for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/60—Methods for processing data by generating or executing the game program
    • A63F2300/63—Methods for processing data by generating or executing the game program for controlling the execution of the game in time
    • A63F2300/638—Methods for processing data by generating or executing the game program for controlling the execution of the game in time according to the timing of operation or a time limit
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/60—Methods for processing data by generating or executing the game program
    • A63F2300/64—Methods for processing data by generating or executing the game program for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/60—Methods for processing data by generating or executing the game program
    • A63F2300/66—Methods for processing data by generating or executing the game program for rendering three dimensional images
    • A63F2300/6607—Methods for processing data by generating or executing the game program for rendering three dimensional images for animating game characters, e.g. skeleton kinematics
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/8029—Fighting without shooting

Definitions

  • the invention relates to human-computer interaction and machine vision.
  • the invention relates to a method, a system and a computer program for instructing and evaluating movement in a computerized movement instruction system.
  • Many movement instruction systems are based on the concept of a timeline so that visual instruc ⁇ tions move linearly on screen along an axis that represents time. For example, in many dancing games the time axis is vertical and there are arrows moving upwards from the bottom of the screen and the player has to step on buttons corresponding to the arrows when the arrows reach the top of the screen.
  • a timeline enables the user to predict what actions are expected in the future, which is crucial to perform complex choreographed moves.
  • Karaoke games such as Sing Star also provide an example of a simple horizontal timeline.
  • the main or only input signal is the pitch of the player's voice. This allows the game to display both the player' s desired and actual action as simple 2d line graphs with the horizontal axis representing time and vertical axis representing pitch.
  • the main problem is the multidimensionality of the data.
  • the player's body configuration can be represented as a real-valued vec- tor b, e.g., with each element corresponding to an angle of a body joint. Displaying the desired behavior b(t) of the body as a function of time using simple graphs would require either more than three coordinate axes, or several graphs, which is not visually intui- tive. More efficient and easy to understand visualiza ⁇ tion systems need to be developed.
  • This invention pro ⁇ vides a solution to instructing time-varying full body movement precisely, predictably and clearly.
  • align is often used, e.g. in the phrase "controlling the appearance of the preview instruction so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction".
  • aligning we mean that predetermined parts of the vis- ual objects of interest (in this case the preview and scrolling instructions) are located on screen within a predetermined distance from each other.
  • align at the detailed description of the invention with references to figures 3 and 4.
  • a first aspect of the present inven ⁇ tion is a method of instructing the user' s movement and poses in a computerized movement instruction sys ⁇ tem. The user's visual representation is displayed. A graphical instruction scrolling on a timeline is dis- played.
  • a graphical preview instruction is displayed.
  • the appearance of the preview instruction is controlled so that it becomes visible before the moment the user should match the scrolling instruction.
  • the appearance of the preview instruction is controlled so that it aligns with the scrolling instruction essentially at the moment the user should match the scroll- ing instruction. It is determined whether the user matches the instructions within a time interval deter ⁇ mined in relation to the moment the preview instruc ⁇ tion and the scrolling instruction align. Feedback indicating whether the user matched the instructions is provided.
  • a second aspect of the present invention is a computerized movement instruction system for instruct ⁇ ing the user's movement and poses.
  • the computerized movement instruction system comprises a screen config- ured to display the user's visual representation.
  • the computerized movement instruction system further comprises a computer configured to generate a graphical instruction scrolling on a timeline and a graphical preview instruction, for display on the screen.
  • the computer is further configured to control the appear ⁇ ance of the preview instruction so that it becomes visible before the moment the user should match the scrolling instruction.
  • the computer is further configured to control the appearance of the preview instruc- tion so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction.
  • the computer is further configured to determine if the user matches the instructions within a time interval determined in relation to the moment the preview instruction and the scrolling instruction align.
  • the computer is further configured to provide feedback that indicates whether the user matched the instructions.
  • a third aspect of the present invention is a computer program for instructing the user' s movement and poses in a computerized movement instruction sys ⁇ tem.
  • the computer program is embodied on a computer- readable medium comprising program code means adapted to perform the following steps when the program is executed in a computing device:
  • the pro ⁇ viding the feedback comprises stopping the scrolling instruction when it aligns with the preview instruc- tion and displaying the stopped instruction until the user has matched it or until a predetermined time has passed .
  • the pro ⁇ viding the feedback comprises increasing the user' s score and making the stopped or scrolling instruction disappear when the user matches the instruction.
  • the determining that the user has matched said instructions comprises detecting that the user' s visual representa ⁇ tion collides with at least one of said instructions, and that the colliding part of the user' s visual rep ⁇ resentation moves in a predetermined direction at a predetermined speed.
  • the user' s visual representation comprises a real-time video im- age of the user captured using a video camera.
  • the preview in ⁇ struction may comprise a circle and the scrolling in ⁇ struction may comprise a human silhouette, and the in ⁇ structions are considered to align when the center of the circle and the right hand of the silhouette are located on screen within a predetermined distance from each other.
  • an example of the player matching the instructions is that the player must move so that the center of the palm of his video image moves inside a circle of the preview instruction half a second before or after a hand of a human silhouette in the scrolling instruction moves to the center of the said circle.
  • the ap- pearance of the preview instruction is controlled so that it becomes visible before the moment the user should match the scrolling instruction.
  • the preview instruction is a static image of a human silhouette and the opacity of the image is controlled so that the image transforms from transparent to opa ⁇ que one second before a scrolling instruction with a similar silhouette aligns with it.
  • this invention provides at the same time more precise and predictable instructions.
  • This invention also provides the user more precise feedback about his or her performance.
  • the combination of dynamically appearing preview instructions, scroll ⁇ ing instructions and the player' s visual display over- laid with the instructions enable one to represent both time and spatial dimensions on the same visual axes.
  • the combination allows one to fully specify the user's desired body configuration (pose) at a prede ⁇ termined time, approximating b(t) in a piecewise man- ner.
  • the combination also allows the user to visually compare his or her current pose to the instructions. All these three elements (space, time, feedback about the user' s full body) are necessary for a movement in ⁇ struction system that does not limit the range of in- structed movements.
  • Fig. 1 is an illustration of an example im- plementation of a system according to the present invention
  • Fig. 2 is an illustration of an example of the graphical output rendered by a system according to the present invention
  • Fig. 3 is a further example of the graphical output rendered by a system according to the present invention .
  • Fig. 4 is a further example of the graphical output rendered by a system according to the present invention.
  • Fig. 5 is a further example of the graphical output rendered by a system according to the present invention .
  • FIG. 1 a typical embodiment of the pre- sent invention is illustrated.
  • the user 10 moves in the vicinity of a sensor device 11, which is a camera in the example illustrated.
  • the computer 15 renders on the screen 12 visual output comprising the user' s vis ⁇ ual representation 13 and other computer graphics 14, for example, the enemies in a fighting game.
  • the us ⁇ er's visual representation 13 may overlap the computer graphics 14, allowing the user to virtually touch and interact with computer generated objects and agents.
  • the user's visual representation 13 may be the play- er's real-time video image captured using the camera 11, possibly processed with image processing algo ⁇ rithms, e.g., to remove the background of the user from the image.
  • the user's visual representation 13 may also be a computer generated 2d or 3d avatar that copies the user's movements based on motion tracking algorithms applied to the data provided by the sensor device 11.
  • the basic principle of the present invention is to show the user of a movement instruction system a plurality of snapshots of the desired body configura ⁇ tions on a timeline.
  • the choreographer may choose a number of body configurations from a sequence of movements.
  • the in ⁇ struction system may then be configured to show visual instructions, e.g., human silhouettes corresponding to the desired body configurations (poses) on a timeline, and also the user' s visual representation superimposed on the timeline.
  • the goal of the user is to match the instructions by aligning or moving his/her visual rep ⁇ resentation with the instructions at predetermined mo ⁇ ments .
  • the abovementioned problems are solved so that the visual instructions are split into two parts, a scrolling instruction and a preview instruction that are configured to align at the moment the user should also align with them or otherwise match the instruc ⁇ tions. From the desired moment of matching we use the term matching time. The user should also move to the place where the scrolling and preview instructions align. This way, the system can instruct both the us ⁇ er' s body configurations and location with respect to time.
  • the scrolling instructions depict body poses and move li ⁇ nearly along the timeline, and the preview instruc- tions stay in the location where the user should match the pose, and only become visible, e.g., one second before the matching time. This minimizes the amount of visual information and maximizes the clarity of pres ⁇ entation .
  • the description is mainly written from the point of view of a movement based game or a movement training game where the computer provides the user pose instructions that he/she must match.
  • the computer gives the user feedback, e.g., with visual and sound effects and by increasing the user's score after successful matching of the poses.
  • Figures 2-4 examples of the visual output of a system according to the present invention are illustrated.
  • Figures 2-4 are in chronological order to illustrate the behavior of the system over time.
  • the user's visual representation 20, 30, 40 is shown to- gether with instruction graphics 21, 22, 23, 31, 32, 33, 34, 35, 36, 37, 39, 41, 42, 43, 44, 45, 46, 47.
  • the instruction graphics move on a timeline so that their position varies as a function of time.
  • the time is mapped to the horizontal coordi- nate axis so that the graphics scroll from right to left.
  • the user's visual representation 20, 30, 40 and the preview instructions 31, 41 do not typ ⁇ ically scroll.
  • the user is shown scrolling in- structions 21, 23 and a movement trail instruction 22 that indicates the desired motion of the user' s hand after matching the pose depicted by the scrolling instruction 21.
  • Such movement trails may be provided for any body part.
  • the movement trails may also indicate the desired motion before matching the pose.
  • FIG 3 time has passed in relation to Figure 2, and the instructions 21, 22, 23 of Figure 2 have scrolled left, now shown as the instructions 32, 37, 33, respectively.
  • the user has assumed the correct pose instructed by the scrolling instruction 32.
  • a preview instruction 31 related to the scrolling instruction 32 is also displayed.
  • the preview instruction shows the location where the user should move, which in the case of Figure 3 means to the left of the user's current position. If the user does not realize that he/she has to move, the system may display additional instructions, e.g., the arrow 39.
  • the system makes it possible to instruct precise rhythm of arbitrary pose sequences - the correct mo- ment to match the pose is when the preview and scroll ⁇ ing instructions 31 and 32 align.
  • the preview instruc ⁇ tions typically work so that they fade in slightly be ⁇ fore the user should match the corresponding scrolling instruction.
  • the preview instructions are not kept visible all the time to reduce visual clutter. If the user is already in the correct location, the preview instruction may not be shown at all.
  • the scrolling instructions may stop and fade out only after the user has matched them (in case of the pose instruction 32, the instruction fades out when the user matches the pose at the correct time and in the correct location) .
  • the system may reward the user of correct matching of poses, for example, with score bo- nuses and audiovisual effects.
  • Embodiments of the present invention may utilize different instruction graphics, depending on the desired movements. Examples are given in Figure 3, including textual instructions 34, forbidden areas 35, and movement targets 36.
  • the forbidden areas work so that, for example, the user's score is decreased or the user cannot match pose instructions while touching a forbidden area.
  • the forbidden area 35 the user should jump over the area to reach and match the movement target 36.
  • Movement targets may be speci ⁇ fied for a particular or any body part, and optionally so that they may only be matched by touching the tar- get from a particular direction on the screen, e.g., using an axe kick where the foot moves downwards.
  • the movement target 36 may be depicted as a brick wall that breaks when the user kicks it using a jump kick high enough to avoid the forbidden area 36.
  • the graphical output of a system according to the present invention may also comprise an additional representation 24, 38, 48 of the user.
  • Figure 4 shows how the additional representation 48 may be manipulated in a time-varying manner, e.g., ro ⁇ tated when instructions are matched to create the il ⁇ lusion of the user performing flamboyant acrobatic stunts although the user is in real life moving in a less risky manner.
  • the additional representation is omitted, but after finishing a movement sequence, the user is provided a replay where his/her recorded and manipu- lated movements are shown, for example a simple jump turning into an air summersault, where the computer moves and rotates the user's visual representation. This way, one can implement a game, where the player becomes a stunt man with the goal of performing cho- reographed stunts for a movie.
  • the user performs the movements on a scroll ⁇ ing, non-rotating timeline.
  • the graphical output also comprises a virtual world that revolves around the us- er, mimicking summersaults where the virtual camera is fixed to the user's torso. After performing the move ⁇ ments, the sequence is shown in a replay where the virtual camera is fixed to the virtual world so that the user is rotated within the virtual world.
  • the user's visual representation 20, 30, 40, 50 may be ma- nipulated directly.
  • suitable manipula ⁇ tion techniques are time-varying rotating, scaling, translating and mirroring of the user' s visual representation.
  • the instruction graphics may be manipulated as well, e.g., so that the timeline rotates in syn- chronization with the user's visual representation.
  • the scrolling instruction 32 may be consid ⁇ ered to align with the similar preview instruction 31 when they are located within a predetermined distance from each other, an example of which is shown in Figure 4, where the said preview instruction is numbered 41 and the said scrolling instruction is numbered 42.
  • the player's visual representation 40 may be consid- ered to align with the scrolling instruction silhouette 42 if the player's hands and feet (or the corre ⁇ sponding locations inferred by a motion tracking sys ⁇ tem) in the video image are within a predetermined distance from the hands and feet of the silhouette.
  • the player may be considered to match the instructions and points may be awarded in the game if the player's visual representation aligns with the scrolling or preview instruction within a predetermined time interval from the moment when the scrolling and preview instructions align.
  • Figure 5 shows the graphical output of an ad- vanced embodiment of the present invention, where the timeline 55 is three-dimensional or isometric. Simi ⁇ larly as in the embodiments above, both preview in ⁇ structions 51 and scrolling instructions 52, 53, 54 are displayed. Forbidden areas 54 are typically also three-dimensional in this case.
  • the us ⁇ er' s movements are typically only instructed in a 2d plane (jumping and moving sideways on a line) .
  • the time axis of the timeline aligns with the user's movements, making location of a graphical in ⁇ struction ambiguous in time and space.
  • a three- dimensional timeline such as the one in Figure 5 may either be used to instruct 2d movement more clearly or to instruct full 3d movement. The latter case requires preview instructions that are located in the 3d space so that they align with scrolling instructions at the correct location and time.
  • An advanced embodiment of the invention may also have multiple simultaneous timelines, so that some instructions are, e.g., scrolling from left to right and some are scrolling from top to bottom. This is particularly useful for forbidden areas.
  • Horizontally scrolling forbidden areas can constrain the us- er's movement in vertical direction, e.g., requiring the user to jump or duct.
  • Vertically scrolling forbid ⁇ den areas can constrain the movement horizontally, in ⁇ structing the user to step aside.
  • An advanced embodiment of the invention may also have a content generation system where the user records a video and then marks the video frames from which the system generates pose instructions, e.g., by using computer vision techniques to extract the out ⁇ line of the user's silhouette.
  • a preview instruction does not need to visu ⁇ ally similar to the scrolling instructions.
  • a preview instruction may only comprise of markers for the desired locations of the user's hands.
  • the scrolling and preview instructions can also be animated. For example, if the user is in ⁇ structed to jump up, the scrolling instruction may first show a person standing in the pose preceding the jump, then play a jump animation and stop in midair in the pose that the player should match. In this case, the preview instruction would also show the pose in midair and appear only after the animation has been played.
  • detecting that the user has matched a pose can be implemented either using a body tracking device, or by simple computer vision methods, e.g., detecting that there is motion at the movement targets at the matching time and no motion within the forbidden area. In addition to motion, one may check whether the user' s silhouette overlaps with the target or forbidden areas.
  • a person skilled in computer vision can typically implement various motion detection and silhouette extraction me ⁇ thods.
  • Complex poses can also be detected with rea ⁇ sonably accuracy using target and forbidden areas that are not necessarily displayed to the user.
  • the exemplary embodiments can include, for example, any suitable servers, workstations, PCs, lap- top computers, personal digital assistants (PDAs) , In ⁇ ternet appliances, handheld devices, cellular tele ⁇ phones, smart phones, wireless devices, game consoles, other devices, and the like, capable of performing the processes of the exemplary embodiments.
  • PDAs personal digital assistants
  • the devices and subsystems of the exemplary embodiments can commu ⁇ nicate with each other using any suitable protocol and can be implemented using one or more programmed com ⁇ puter systems or devices.
  • One or more interface mechanisms can be used with the exemplary embodiments, including, for example, Internet access, telecommunications in any suit ⁇ able form (e.g., voice, modem, and the like), wireless communications media, and the like.
  • em ⁇ ployed communications networks or links can include one or more wireless communications networks, cellular communications networks, 3G communications networks, Public Switched Telephone Network (PSTNs) , Packet Data Networks (PDNs) , the Internet, intranets, a combina ⁇ tion thereof, and the like.
  • PSTNs Public Switched Telephone Network
  • PDNs Packet Data Networks
  • the exemplary em ⁇ bodiments are for exemplary purposes, as many varia ⁇ tions of the specific hardware used to implement the exemplary embodiments are possible, as will be appre ⁇ ciated by those skilled in the hardware and/or soft- ware art(s) .
  • the functionality of one or more of the components of the exemplary embodiments can be implemented via one or more hardware and/or software devices.
  • the exemplary embodiments can store informa- tion relating to various processes described herein.
  • This information can be stored in one or more memo ⁇ ries, such as a hard disk, optical disk, magneto- optical disk, RAM, and the like.
  • One or more databases can store the information used to implement the exem ⁇ plary embodiments of the present inventions.
  • the data ⁇ bases can be organized using data structures (e.g., records, tables, arrays, fields, graphs, trees, lists, and the like) included in one or more memories or sto ⁇ rage devices listed herein.
  • the processes described with respect to the exemplary embodiments can include appropriate data structures for storing data collected and/or generated by the processes of the devices and subsystems of the exemplary embodiments in one or more databases .
  • All or a portion of the exemplary embodiments can be conveniently implemented using one or more gen- eral purpose processors, microprocessors, digital sig ⁇ nal processors, micro-controllers, and the like, pro ⁇ grammed according to the teachings of the exemplary embodiments of the present inventions, as will be ap ⁇ preciated by those skilled in the computer and/or software art(s) .
  • Appropriate software can be readily prepared by programmers of ordinary skill based on the teachings of the exemplary embodiments, as will be ap ⁇ preciated by those skilled in the software art.
  • the exemplary embodiments can be implemented by the preparation of application-specific integrated circuits or by interconnecting an appropriate network of conventional component circuits, as will be appre ⁇ ciated by those skilled in the electrical art(s) .
  • the exemplary embodiments are not limited to any specific combination of hardware and/or software.
  • the exemplary embodiments of the present inventions can include software for control ⁇ ling the components of the exemplary embodiments, for driving the components of the exemplary embodiments, for enabling the components of the exemplary embodi ⁇ ments to interact with a human user, and the like.
  • Such software can include, but is not limited to, de ⁇ vice drivers, firmware, operating systems, development tools, applications software, and the like.
  • Such com ⁇ puter readable media further can include the computer program product of an embodiment of the present inven ⁇ tions for performing all or a portion (if processing is distributed) of the processing performed in imple ⁇ menting the inventions.
  • Computer code devices of the exemplary embodiments of the present inventions can include any suitable interpretable or executable code mechanism, including but not limited to scripts, in ⁇ terpretable programs, dynamic link libraries (DLLs) , Java classes and applets, complete executable pro ⁇ grams, Common Object Request Broker Architecture (COR- BA) objects, and the like. Moreover, parts of the processing of the exemplary embodiments of the present inventions can be distributed for better performance, reliability, cost, and the like.
  • DLLs dynamic link libraries
  • Java classes and applets Java classes and applets
  • COR- BA Common Object Request Broker Architecture
  • the components of the exem- plary embodiments can include computer readable medium or memories for holding instructions programmed ac ⁇ cording to the teachings of the present inventions and for holding data structures, tables, records, and/or other data described herein.
  • Computer readable medium can include any suitable medium that participates in providing instructions to a processor for execution. Such a medium can take many forms, including but not limited to, non-volatile media, volatile media, trans ⁇ mission media, and the like.
  • Non-volatile media can include, for example, optical or magnetic disks, mag ⁇ neto-optical disks, and the like.
  • Volatile media can include dynamic memories, and the like.
  • Transmission media can include coaxial cables, copper wire, fiber optics, and the like.
  • Transmission media also can take the form of acoustic, optical, electromagnetic waves, and the like, such as those generated during radio frequency (RF) communications, infrared (IR) data com- munications, and the like.
  • Common forms of computer- readable media can include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other suitable magnetic medium, a CD-ROM, CD ⁇ R, CD ⁇ RW, DVD, DVD-RAM, DVD1RW, DVD ⁇ R, HD DVD, HD DVD-R, HD DVD- RW, HD DVD-RAM, Blu-ray Disc, any other suitable opti ⁇ cal medium, punch cards, paper tape, optical mark sheets, any other suitable physical medium with pat ⁇ terns of holes or other optically recognizable indi- cia, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other suitable memory chip or cartridge, a carrier wave or any other suitable medium from which a computer can read .

Abstract

The invention allows instructing the user' s movement and poses in a computerized movement instruction system. The user' s visual representation, a graphical instruction scrolling on a timeline, and a graphical preview instruction are displayed. The appearance of the preview instruction is controlled so that it becomes visible before the moment the user should match the scrolling instruction, and so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction. It is determined whether the user matches the instructions within a time interval determined in relation to the moment the preview instruction and the scrolling instruction align. Feedback indicating whether the user matched the instructions is provided.

Description

A METHOD, A SYSTEM AND A COMPUTER PROGRAM FOR INSTRUCTING AND EVALUATING FULL-BODY MOVEMENT
FIELD OF THE INVENTION
The invention relates to human-computer interaction and machine vision. In particular, the invention relates to a method, a system and a computer program for instructing and evaluating movement in a computerized movement instruction system.
BACKGROUND OF THE INVENTION
There are many computerized movement in¬ struction systems and games, e.g., dancing games. How¬ ever, the movements are usually simplified because in¬ structing and judging full body movement is a diffi¬ cult communication and analysis problem.
Many movement instruction systems are based on the concept of a timeline so that visual instruc¬ tions move linearly on screen along an axis that represents time. For example, in many dancing games the time axis is vertical and there are arrows moving upwards from the bottom of the screen and the player has to step on buttons corresponding to the arrows when the arrows reach the top of the screen. There are also other approaches, e.g., the Just Follow Me vir¬ tual reality movement instruction system, where the teacher' s visual 3d representation overlaps with the student body in the virtual reality and the student's goal is to align his or her body with the teacher. However, a timeline enables the user to predict what actions are expected in the future, which is crucial to perform complex choreographed moves. Karaoke games such as Sing Star also provide an example of a simple horizontal timeline. In the games, the main or only input signal is the pitch of the player's voice. This allows the game to display both the player' s desired and actual action as simple 2d line graphs with the horizontal axis representing time and vertical axis representing pitch.
SUMMARY
In instructing full-body movement in an interactive manner, e.g., in a dancing game or a choreo¬ graphed martial arts game, the main problem is the multidimensionality of the data. The player's body configuration can be represented as a real-valued vec- tor b, e.g., with each element corresponding to an angle of a body joint. Displaying the desired behavior b(t) of the body as a function of time using simple graphs would require either more than three coordinate axes, or several graphs, which is not visually intui- tive. More efficient and easy to understand visualiza¬ tion systems need to be developed. This invention pro¬ vides a solution to instructing time-varying full body movement precisely, predictably and clearly.
In this document, the term "align" is often used, e.g. in the phrase "controlling the appearance of the preview instruction so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction". By aligning, we mean that predetermined parts of the vis- ual objects of interest (in this case the preview and scrolling instructions) are located on screen within a predetermined distance from each other. Please see further explanation for the term "align" at the detailed description of the invention with references to figures 3 and 4. A first aspect of the present inven¬ tion is a method of instructing the user' s movement and poses in a computerized movement instruction sys¬ tem. The user's visual representation is displayed. A graphical instruction scrolling on a timeline is dis- played. A graphical preview instruction is displayed. The appearance of the preview instruction is controlled so that it becomes visible before the moment the user should match the scrolling instruction. The appearance of the preview instruction is controlled so that it aligns with the scrolling instruction essentially at the moment the user should match the scroll- ing instruction. It is determined whether the user matches the instructions within a time interval deter¬ mined in relation to the moment the preview instruc¬ tion and the scrolling instruction align. Feedback indicating whether the user matched the instructions is provided.
A second aspect of the present invention is a computerized movement instruction system for instruct¬ ing the user's movement and poses. The computerized movement instruction system comprises a screen config- ured to display the user's visual representation. The computerized movement instruction system further comprises a computer configured to generate a graphical instruction scrolling on a timeline and a graphical preview instruction, for display on the screen. The computer is further configured to control the appear¬ ance of the preview instruction so that it becomes visible before the moment the user should match the scrolling instruction. The computer is further configured to control the appearance of the preview instruc- tion so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction. The computer is further configured to determine if the user matches the instructions within a time interval determined in relation to the moment the preview instruction and the scrolling instruction align. The computer is further configured to provide feedback that indicates whether the user matched the instructions.
A third aspect of the present invention is a computer program for instructing the user' s movement and poses in a computerized movement instruction sys¬ tem. The computer program is embodied on a computer- readable medium comprising program code means adapted to perform the following steps when the program is executed in a computing device:
displaying the user's visual representation; displaying a graphical instruction scrolling on a timeline;
displaying a graphical preview instruction; controlling the appearance of the preview instruction so that it becomes visible before the moment the user should match the scrolling instruction;
controlling the appearance of the preview instruction so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction;
determining whether the user matches the instructions within a time interval determined in rela¬ tion to the moment the preview instruction and the scrolling instruction align; and
providing feedback that indicates whether the user matched the instructions.
In an embodiment of the invention, the determining whether the user matches the instructions com¬ prises detecting that the hands and feet of the user' s visual representation essentially align with the loca- tions of hands and feet illustrated by the instruc¬ tions .
In an embodiment of the invention, the pro¬ viding the feedback comprises stopping the scrolling instruction when it aligns with the preview instruc- tion and displaying the stopped instruction until the user has matched it or until a predetermined time has passed .
In an embodiment of the invention, the pro¬ viding the feedback comprises increasing the user' s score and making the stopped or scrolling instruction disappear when the user matches the instruction. In an embodiment of the invention, the determining that the user has matched said instructions comprises detecting that the user' s visual representa¬ tion collides with at least one of said instructions, and that the colliding part of the user' s visual rep¬ resentation moves in a predetermined direction at a predetermined speed.
In an embodiment of the invention, the user' s visual representation comprises a real-time video im- age of the user captured using a video camera.
In an alternative embodiment, the preview in¬ struction may comprise a circle and the scrolling in¬ struction may comprise a human silhouette, and the in¬ structions are considered to align when the center of the circle and the right hand of the silhouette are located on screen within a predetermined distance from each other. In this case, an example of the player matching the instructions is that the player must move so that the center of the palm of his video image moves inside a circle of the preview instruction half a second before or after a hand of a human silhouette in the scrolling instruction moves to the center of the said circle.
In an embodiment of the invention, the ap- pearance of the preview instruction is controlled so that it becomes visible before the moment the user should match the scrolling instruction. In graphics programming terms, a typical example of this is that the preview instruction is a static image of a human silhouette and the opacity of the image is controlled so that the image transforms from transparent to opa¬ que one second before a scrolling instruction with a similar silhouette aligns with it.
Compared to previous movement instruction systems, e.g., dancing games that display moving ar¬ rows on a timeline, this invention provides at the same time more precise and predictable instructions. This invention also provides the user more precise feedback about his or her performance. The combination of dynamically appearing preview instructions, scroll¬ ing instructions and the player' s visual display over- laid with the instructions enable one to represent both time and spatial dimensions on the same visual axes. The combination allows one to fully specify the user's desired body configuration (pose) at a prede¬ termined time, approximating b(t) in a piecewise man- ner. The combination also allows the user to visually compare his or her current pose to the instructions. All these three elements (space, time, feedback about the user' s full body) are necessary for a movement in¬ struction system that does not limit the range of in- structed movements.
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are included to provide a further understanding of the invention and constitute a part of this specification, illus¬ trate embodiments of the invention and together with the description help to explain the principles of the invention. In the drawings:
Fig. 1 is an illustration of an example im- plementation of a system according to the present invention,
Fig. 2 is an illustration of an example of the graphical output rendered by a system according to the present invention,
Fig. 3 is a further example of the graphical output rendered by a system according to the present invention .
Fig. 4 is a further example of the graphical output rendered by a system according to the present invention. Fig. 5 is a further example of the graphical output rendered by a system according to the present invention . DETAILED DESCRIPTION OF THE INVENTION
Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings.
In Figure 1, a typical embodiment of the pre- sent invention is illustrated. The user 10 moves in the vicinity of a sensor device 11, which is a camera in the example illustrated. The computer 15 renders on the screen 12 visual output comprising the user' s vis¬ ual representation 13 and other computer graphics 14, for example, the enemies in a fighting game. The us¬ er's visual representation 13 may overlap the computer graphics 14, allowing the user to virtually touch and interact with computer generated objects and agents. The user's visual representation 13 may be the play- er's real-time video image captured using the camera 11, possibly processed with image processing algo¬ rithms, e.g., to remove the background of the user from the image. The user's visual representation 13 may also be a computer generated 2d or 3d avatar that copies the user's movements based on motion tracking algorithms applied to the data provided by the sensor device 11.
The basic principle of the present invention is to show the user of a movement instruction system a plurality of snapshots of the desired body configura¬ tions on a timeline. In an embodiment of the inven¬ tion, the choreographer may choose a number of body configurations from a sequence of movements. The in¬ struction system may then be configured to show visual instructions, e.g., human silhouettes corresponding to the desired body configurations (poses) on a timeline, and also the user' s visual representation superimposed on the timeline. The goal of the user is to match the instructions by aligning or moving his/her visual rep¬ resentation with the instructions at predetermined mo¬ ments .
However, displaying the user and scrolling instruction graphic on a timeline is not enough, be¬ cause one needs to instruct both time and location in space. For example, most martial arts choreographies consist of both steps and poses so that the poses should be matched at the right time and in the correct location in space. Time adds a fourth dimension to 3d space, so that when instructing three-dimensional movement as a function of time with 3d graphics, one has to show both time and some position variable on the same axis. Because of this ambiguity the user may not know if he/she is supposed to move towards a scrolling instruction or to wait until the instruction comes closer. Furthermore, in many cases it would be desirable to use only 2d graphics to make the overall visual appearance simpler.
In a method and system according to this invention, the abovementioned problems are solved so that the visual instructions are split into two parts, a scrolling instruction and a preview instruction that are configured to align at the moment the user should also align with them or otherwise match the instruc¬ tions. From the desired moment of matching we use the term matching time. The user should also move to the place where the scrolling and preview instructions align. This way, the system can instruct both the us¬ er' s body configurations and location with respect to time. In a typical embodiment of the invention, the scrolling instructions depict body poses and move li¬ nearly along the timeline, and the preview instruc- tions stay in the location where the user should match the pose, and only become visible, e.g., one second before the matching time. This minimizes the amount of visual information and maximizes the clarity of pres¬ entation .
In the following, the description is mainly written from the point of view of a movement based game or a movement training game where the computer provides the user pose instructions that he/she must match. The computer gives the user feedback, e.g., with visual and sound effects and by increasing the user's score after successful matching of the poses.
In Figures 2-4, examples of the visual output of a system according to the present invention are illustrated. Figures 2-4 are in chronological order to illustrate the behavior of the system over time. The user's visual representation 20, 30, 40 is shown to- gether with instruction graphics 21, 22, 23, 31, 32, 33, 34, 35, 36, 37, 39, 41, 42, 43, 44, 45, 46, 47. The instruction graphics move on a timeline so that their position varies as a function of time. In Figures 2-4, the time is mapped to the horizontal coordi- nate axis so that the graphics scroll from right to left. However, the user's visual representation 20, 30, 40 and the preview instructions 31, 41 do not typ¬ ically scroll.
In Figure 2, the user is shown scrolling in- structions 21, 23 and a movement trail instruction 22 that indicates the desired motion of the user' s hand after matching the pose depicted by the scrolling instruction 21. Such movement trails may be provided for any body part. The movement trails may also indicate the desired motion before matching the pose.
In Figure 3, time has passed in relation to Figure 2, and the instructions 21, 22, 23 of Figure 2 have scrolled left, now shown as the instructions 32, 37, 33, respectively. As illustrated by the user's visual representation 30, the user has assumed the correct pose instructed by the scrolling instruction 32. To further clarify where and when the user should assume the pose, a preview instruction 31 related to the scrolling instruction 32 is also displayed. The preview instruction shows the location where the user should move, which in the case of Figure 3 means to the left of the user's current position. If the user does not realize that he/she has to move, the system may display additional instructions, e.g., the arrow 39. The system makes it possible to instruct precise rhythm of arbitrary pose sequences - the correct mo- ment to match the pose is when the preview and scroll¬ ing instructions 31 and 32 align. The preview instruc¬ tions typically work so that they fade in slightly be¬ fore the user should match the corresponding scrolling instruction. The preview instructions are not kept visible all the time to reduce visual clutter. If the user is already in the correct location, the preview instruction may not be shown at all.
After aligning with the preview instruction, the scrolling instructions may stop and fade out only after the user has matched them (in case of the pose instruction 32, the instruction fades out when the user matches the pose at the correct time and in the correct location) . The system may reward the user of correct matching of poses, for example, with score bo- nuses and audiovisual effects.
Embodiments of the present invention may utilize different instruction graphics, depending on the desired movements. Examples are given in Figure 3, including textual instructions 34, forbidden areas 35, and movement targets 36. The forbidden areas work so that, for example, the user's score is decreased or the user cannot match pose instructions while touching a forbidden area. In case of the forbidden area 35, the user should jump over the area to reach and match the movement target 36. Movement targets may be speci¬ fied for a particular or any body part, and optionally so that they may only be matched by touching the tar- get from a particular direction on the screen, e.g., using an axe kick where the foot moves downwards. In case of a martial arts training system, the movement target 36 may be depicted as a brick wall that breaks when the user kicks it using a jump kick high enough to avoid the forbidden area 36.
In Figure 4, time has passed in relation to Figure 3. The instructions 41, 42, 43, 44, 45, 46, 47 correspond to instructions 31, 32, 33, 34, 35, 36, 37 in Figure 3, respectively. The preview instruction 41 and scrolling instruction 42 now align and the user 40 has moved into the correct location.
Optionally, the graphical output of a system according to the present invention may also comprise an additional representation 24, 38, 48 of the user. Figure 4 shows how the additional representation 48 may be manipulated in a time-varying manner, e.g., ro¬ tated when instructions are matched to create the il¬ lusion of the user performing flamboyant acrobatic stunts although the user is in real life moving in a less risky manner. In an alternative embodiment of the invention, the additional representation is omitted, but after finishing a movement sequence, the user is provided a replay where his/her recorded and manipu- lated movements are shown, for example a simple jump turning into an air summersault, where the computer moves and rotates the user's visual representation. This way, one can implement a game, where the player becomes a stunt man with the goal of performing cho- reographed stunts for a movie.
In an advanced embodiment of the present in¬ vention, the user performs the movements on a scroll¬ ing, non-rotating timeline. The graphical output also comprises a virtual world that revolves around the us- er, mimicking summersaults where the virtual camera is fixed to the user's torso. After performing the move¬ ments, the sequence is shown in a replay where the virtual camera is fixed to the virtual world so that the user is rotated within the virtual world.
In another embodiment of the invention, the user's visual representation 20, 30, 40, 50 may be ma- nipulated directly. Examples of the suitable manipula¬ tion techniques are time-varying rotating, scaling, translating and mirroring of the user' s visual representation. The instruction graphics may be manipulated as well, e.g., so that the timeline rotates in syn- chronization with the user's visual representation.
The term "align" can be understood more clearly from the following explanation with references to figures 3 and 4. Consider the embodiment of the present invention illustrated in Figures 3 and 4, where the player's visual representation on screen 30 is the player's real-time video image captured by an imaging device, the scrolling instructions 32, 33, 34, 35, 36, 37 comprise the freeze frame silhouettes 32,33 of a human performing the instructed choreography, the preview instructions 31 comprise the same silhouettes displayed one at a time, and both instructions are shown overlaid on top of the player's video image. In this case, the scrolling instruction 32 may be consid¬ ered to align with the similar preview instruction 31 when they are located within a predetermined distance from each other, an example of which is shown in Figure 4, where the said preview instruction is numbered 41 and the said scrolling instruction is numbered 42. The player's visual representation 40 may be consid- ered to align with the scrolling instruction silhouette 42 if the player's hands and feet (or the corre¬ sponding locations inferred by a motion tracking sys¬ tem) in the video image are within a predetermined distance from the hands and feet of the silhouette. Furthermore, the player may be considered to match the instructions and points may be awarded in the game if the player's visual representation aligns with the scrolling or preview instruction within a predetermined time interval from the moment when the scrolling and preview instructions align.
Figure 5 shows the graphical output of an ad- vanced embodiment of the present invention, where the timeline 55 is three-dimensional or isometric. Simi¬ larly as in the embodiments above, both preview in¬ structions 51 and scrolling instructions 52, 53, 54 are displayed. Forbidden areas 54 are typically also three-dimensional in this case. In embodiments of the present invention illustrated in Figures 2-4, the us¬ er' s movements are typically only instructed in a 2d plane (jumping and moving sideways on a line) . In this case, the time axis of the timeline aligns with the user's movements, making location of a graphical in¬ struction ambiguous in time and space. However, using the preview instructions solves this problem. A three- dimensional timeline such as the one in Figure 5 may either be used to instruct 2d movement more clearly or to instruct full 3d movement. The latter case requires preview instructions that are located in the 3d space so that they align with scrolling instructions at the correct location and time.
An advanced embodiment of the invention may also have multiple simultaneous timelines, so that some instructions are, e.g., scrolling from left to right and some are scrolling from top to bottom. This is particularly useful for forbidden areas. Horizontally scrolling forbidden areas can constrain the us- er's movement in vertical direction, e.g., requiring the user to jump or duct. Vertically scrolling forbid¬ den areas can constrain the movement horizontally, in¬ structing the user to step aside.
An advanced embodiment of the invention may also have a content generation system where the user records a video and then marks the video frames from which the system generates pose instructions, e.g., by using computer vision techniques to extract the out¬ line of the user's silhouette.
The preview instructions do not need to visu¬ ally similar to the scrolling instructions. For exam- pie, a preview instruction may only comprise of markers for the desired locations of the user's hands.
The scrolling and preview instructions can also be animated. For example, if the user is in¬ structed to jump up, the scrolling instruction may first show a person standing in the pose preceding the jump, then play a jump animation and stop in midair in the pose that the player should match. In this case, the preview instruction would also show the pose in midair and appear only after the animation has been played.
In an embodiment of the invention, detecting that the user has matched a pose can be implemented either using a body tracking device, or by simple computer vision methods, e.g., detecting that there is motion at the movement targets at the matching time and no motion within the forbidden area. In addition to motion, one may check whether the user' s silhouette overlaps with the target or forbidden areas. A person skilled in computer vision can typically implement various motion detection and silhouette extraction me¬ thods. Complex poses can also be detected with rea¬ sonably accuracy using target and forbidden areas that are not necessarily displayed to the user. In recent years, computer vision human tracking has also im- proved to the point where the player's full body con¬ figuration can be estimated by fitting a skeleton model to the camera view, especially if a 3d camera is used. 3d cameras measure the distance of each pixel in addition to color.
Each of the various elements and embodiments of the invention described above may be implemented in software, in hardware, or as a combination of software and hardware .
The exemplary embodiments can include, for example, any suitable servers, workstations, PCs, lap- top computers, personal digital assistants (PDAs) , In¬ ternet appliances, handheld devices, cellular tele¬ phones, smart phones, wireless devices, game consoles, other devices, and the like, capable of performing the processes of the exemplary embodiments. The devices and subsystems of the exemplary embodiments can commu¬ nicate with each other using any suitable protocol and can be implemented using one or more programmed com¬ puter systems or devices.
One or more interface mechanisms can be used with the exemplary embodiments, including, for example, Internet access, telecommunications in any suit¬ able form (e.g., voice, modem, and the like), wireless communications media, and the like. For example, em¬ ployed communications networks or links can include one or more wireless communications networks, cellular communications networks, 3G communications networks, Public Switched Telephone Network (PSTNs) , Packet Data Networks (PDNs) , the Internet, intranets, a combina¬ tion thereof, and the like.
It is to be understood that the exemplary em¬ bodiments are for exemplary purposes, as many varia¬ tions of the specific hardware used to implement the exemplary embodiments are possible, as will be appre¬ ciated by those skilled in the hardware and/or soft- ware art(s) . For example, the functionality of one or more of the components of the exemplary embodiments can be implemented via one or more hardware and/or software devices.
The exemplary embodiments can store informa- tion relating to various processes described herein. This information can be stored in one or more memo¬ ries, such as a hard disk, optical disk, magneto- optical disk, RAM, and the like. One or more databases can store the information used to implement the exem¬ plary embodiments of the present inventions. The data¬ bases can be organized using data structures (e.g., records, tables, arrays, fields, graphs, trees, lists, and the like) included in one or more memories or sto¬ rage devices listed herein. The processes described with respect to the exemplary embodiments can include appropriate data structures for storing data collected and/or generated by the processes of the devices and subsystems of the exemplary embodiments in one or more databases .
All or a portion of the exemplary embodiments can be conveniently implemented using one or more gen- eral purpose processors, microprocessors, digital sig¬ nal processors, micro-controllers, and the like, pro¬ grammed according to the teachings of the exemplary embodiments of the present inventions, as will be ap¬ preciated by those skilled in the computer and/or software art(s) . Appropriate software can be readily prepared by programmers of ordinary skill based on the teachings of the exemplary embodiments, as will be ap¬ preciated by those skilled in the software art. In ad¬ dition, the exemplary embodiments can be implemented by the preparation of application-specific integrated circuits or by interconnecting an appropriate network of conventional component circuits, as will be appre¬ ciated by those skilled in the electrical art(s) . Thus, the exemplary embodiments are not limited to any specific combination of hardware and/or software.
Stored on any one or on a combination of computer readable media, the exemplary embodiments of the present inventions can include software for control¬ ling the components of the exemplary embodiments, for driving the components of the exemplary embodiments, for enabling the components of the exemplary embodi¬ ments to interact with a human user, and the like. Such software can include, but is not limited to, de¬ vice drivers, firmware, operating systems, development tools, applications software, and the like. Such com¬ puter readable media further can include the computer program product of an embodiment of the present inven¬ tions for performing all or a portion (if processing is distributed) of the processing performed in imple¬ menting the inventions. Computer code devices of the exemplary embodiments of the present inventions can include any suitable interpretable or executable code mechanism, including but not limited to scripts, in¬ terpretable programs, dynamic link libraries (DLLs) , Java classes and applets, complete executable pro¬ grams, Common Object Request Broker Architecture (COR- BA) objects, and the like. Moreover, parts of the processing of the exemplary embodiments of the present inventions can be distributed for better performance, reliability, cost, and the like.
As stated above, the components of the exem- plary embodiments can include computer readable medium or memories for holding instructions programmed ac¬ cording to the teachings of the present inventions and for holding data structures, tables, records, and/or other data described herein. Computer readable medium can include any suitable medium that participates in providing instructions to a processor for execution. Such a medium can take many forms, including but not limited to, non-volatile media, volatile media, trans¬ mission media, and the like. Non-volatile media can include, for example, optical or magnetic disks, mag¬ neto-optical disks, and the like. Volatile media can include dynamic memories, and the like. Transmission media can include coaxial cables, copper wire, fiber optics, and the like. Transmission media also can take the form of acoustic, optical, electromagnetic waves, and the like, such as those generated during radio frequency (RF) communications, infrared (IR) data com- munications, and the like. Common forms of computer- readable media can include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other suitable magnetic medium, a CD-ROM, CD±R, CD±RW, DVD, DVD-RAM, DVD1RW, DVD±R, HD DVD, HD DVD-R, HD DVD- RW, HD DVD-RAM, Blu-ray Disc, any other suitable opti¬ cal medium, punch cards, paper tape, optical mark sheets, any other suitable physical medium with pat¬ terns of holes or other optically recognizable indi- cia, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other suitable memory chip or cartridge, a carrier wave or any other suitable medium from which a computer can read .
It is obvious to a person skilled in the art that with the advancement of technology, the basic idea of the invention may be implemented in various ways. The invention and its embodiments are thus not limited to the examples described above; instead they may vary within the scope of the claims.

Claims

1. A method of instructing the user's move¬ ment and poses in a computerized movement instruction system, c h a r a c t e r i z e d in that the method com- prises:
displaying the user's visual representation;
displaying a graphical instruction scrolling on a timeline ;
displaying a graphical preview instruction;
controlling the appearance of the preview instruc¬ tion so that it becomes visible before the moment the user should match the scrolling instruction;
controlling the appearance of the preview instruc¬ tion so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction;
determining whether the user matches the instructions within a time interval determined in relation to the moment the preview instruction and the scrolling instruction align; and
providing feedback that indicates whether the user matched the instructions.
2. The method according to claim 1, wherein said determining whether the user matches the instruc- tions comprises detecting that the hands and feet of the user' s visual representation essentially align with the locations of hands and feet illustrated by the instructions.
3. The method according to any of claims 1 - 2, wherein said providing the feedback comprises stopping the scrolling instruction when it aligns with the preview instruction and displaying the stopped instruction until the user has matched it or until a predetermined time has passed.
4. The method according to any of claims 1 -
3, wherein said providing the feedback comprises increasing the user' s score and making the stopped or scrolling instruction disappear when the user matches the instruction.
5. The method according to any of claims 1 -
4, wherein said determining that the user has matched said instructions comprises detecting that the user' s visual representation collides with at least one of said instructions, and that the colliding part of the user' s visual representation moves in a predetermined direction at a predetermined speed.
6. The method according to any of claims 1 -
5, wherein the user's visual representation comprises a real-time video image of the user captured using a video camera.
7. A computerized movement instruction system for instructing the user's movement and poses, c h a ¬ r a c t e r i z e d in that the system comprises:
a screen (12) configured to display the user's (10) visual representation (13, 20, 30, 40, 50); and a computer (15) configured to generate a graphical instruction (21, 22, 23, 32, 33, 34, 35, 36, 37, 39, 42, 43, 44, 45, 46, 47, 52, 53, 54) scrolling on a timeline and a graphical preview instruction (31, 41, 51), for display on said screen (12);
wherein said computer (15) is further configured to control the appearance of the preview instruction so that it becomes visible before the moment the user should match the scrolling instruction;
said computer (15) is further configured to con¬ trol the appearance of the preview instruction so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling in¬ struction;
said computer (15) is further configured to deter¬ mine if the user matches the instructions within a time interval determined in relation to the moment the preview instruction and the scrolling instruction align; and said computer (15) is further configured to pro¬ vide feedback that indicates whether the user matched the instructions.
8. The computerized movement instruction sys- tern according to claim 7, wherein said computer (15) is further configured to perform said determining whether the user matches the instructions by detecting that the hands and feet of the user' s visual represen¬ tation essentially align with the locations of hands and feet illustrated by the instructions.
9. The computerized movement instruction sys¬ tem according to any of claims 7 - 8, wherein said computer (15) is further configured to perform said providing the feedback by stopping the scrolling in- struction when it aligns with the preview instruction and displaying the stopped instruction until the user has matched it or until a predetermined time has passed .
10. The computerized movement instruction system according to any of claims 7 - 9, wherein said computer (15) is further configured to perform said providing the feedback by increasing the user' s score and making the stopped or scrolling instruction disappear when the user matches the instruction.
11. The computerized movement instruction system according to any of claims 7 - 10, wherein said computer (15) is further configured to perform said determining that the user has matched said instruc¬ tions by detecting that the user' s visual representa- tion collides with at least one of said instructions, and that the colliding part of the user' s visual rep¬ resentation moves in a predetermined direction at a predetermined speed.
12. The computerized movement instruction system according to any of claims 7 - 11, wherein the user's (10) visual representation (13, 20, 30, 40, 50) comprises a real-time video image of the user (10) captured using a video camera (11) .
13. A computer program for instructing the user's movement and poses in a computerized movement instruction system, c h a r a c t e r i z e d in that the computer program is embodied on a computer-readable medium comprising program code means adapted to perform the following steps when the program is executed in a computing device:
displaying the user's visual representation;
displaying a graphical instruction scrolling on a timeline ;
displaying a graphical preview instruction;
controlling the appearance of the preview instruc- tion so that it becomes visible before the moment the user should match the scrolling instruction;
controlling the appearance of the preview instruc¬ tion so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction;
determining whether the user matches the instructions within a time interval determined in relation to the moment the preview instruction and the scrolling instruction align; and
providing feedback that indicates whether the user matched the instructions.
14. The computer program according to claim 13, wherein said determining whether the user matches the instructions comprises detecting that the hands and feet of the user's visual representation essen¬ tially align with the locations of hands and feet il¬ lustrated by the instructions.
15. The computer program according to any of claims 13 - 14, wherein said providing the feedback comprises stopping the scrolling instruction when it aligns with the preview instruction and displaying the stopped instruction until the user has matched it or until a predetermined time has passed.
16. The computer program according to any of claims 13 - 15, wherein said providing the feedback comprises increasing the user' s score and making the stopped or scrolling instruction disappear when the user matches the instruction.
17. The computer program according to any of claims 13 - 16, wherein said determining that the user has matched said instructions comprises detecting that the user' s visual representation collides with at least one of said instructions, and that the colliding part of the user' s visual representation moves in a predetermined direction at a predetermined speed.
PCT/FI2010/050990 2009-12-02 2010-12-02 A method, a system and a computer program for instructing and evaluating full-body movement WO2011067469A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FI20096276A FI20096276A0 (en) 2009-12-02 2009-12-02 A method, system, and computer program for controlling and evaluating whole body movements
FI20096276 2009-12-02

Publications (1)

Publication Number Publication Date
WO2011067469A1 true WO2011067469A1 (en) 2011-06-09

Family

ID=41462742

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2010/050990 WO2011067469A1 (en) 2009-12-02 2010-12-02 A method, a system and a computer program for instructing and evaluating full-body movement

Country Status (2)

Country Link
FI (1) FI20096276A0 (en)
WO (1) WO2011067469A1 (en)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8439733B2 (en) 2007-06-14 2013-05-14 Harmonix Music Systems, Inc. Systems and methods for reinstating a player within a rhythm-action game
US8444464B2 (en) 2010-06-11 2013-05-21 Harmonix Music Systems, Inc. Prompting a player of a dance game
US8449360B2 (en) 2009-05-29 2013-05-28 Harmonix Music Systems, Inc. Displaying song lyrics and vocal cues
US8465366B2 (en) 2009-05-29 2013-06-18 Harmonix Music Systems, Inc. Biasing a musical performance input to a part
US8550908B2 (en) 2010-03-16 2013-10-08 Harmonix Music Systems, Inc. Simulating musical instruments
US8663013B2 (en) 2008-07-08 2014-03-04 Harmonix Music Systems, Inc. Systems and methods for simulating a rock band experience
US8678896B2 (en) 2007-06-14 2014-03-25 Harmonix Music Systems, Inc. Systems and methods for asynchronous band interaction in a rhythm action game
US8686269B2 (en) 2006-03-29 2014-04-01 Harmonix Music Systems, Inc. Providing realistic interaction to a player of a music-based video game
US8702485B2 (en) 2010-06-11 2014-04-22 Harmonix Music Systems, Inc. Dance game and tutorial
US9024166B2 (en) 2010-09-09 2015-05-05 Harmonix Music Systems, Inc. Preventing subtractive track separation
GB2524402A (en) * 2014-03-20 2015-09-23 2Mee Ltd Augmented Reality Apparatus and Method
US9358456B1 (en) 2010-06-11 2016-06-07 Harmonix Music Systems, Inc. Dance competition game
US9981193B2 (en) 2009-10-27 2018-05-29 Harmonix Music Systems, Inc. Movement based recognition and evaluation
WO2018198909A1 (en) * 2017-04-28 2018-11-01 株式会社ソニー・インタラクティブエンタテインメント Information processing device, information processing method, and program
CN109248414A (en) * 2018-09-30 2019-01-22 深圳市科迈爱康科技有限公司 Training based reminding method, device, equipment and readable storage medium storing program for executing
US10357714B2 (en) 2009-10-27 2019-07-23 Harmonix Music Systems, Inc. Gesture-based user interface for navigating a menu
US10679413B2 (en) 2014-06-10 2020-06-09 2Mee Ltd Augmented reality apparatus and method
US10885333B2 (en) 2012-09-12 2021-01-05 2Mee Ltd Augmented reality apparatus and method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1029565A2 (en) * 1999-02-16 2000-08-23 Konami Co., Ltd. Music staging game apparatus, music staging game method, and readable storage medium
US20020019258A1 (en) * 2000-05-31 2002-02-14 Kim Gerard Jounghyun Methods and apparatus of displaying and evaluating motion data in a motion game apparatus
US20020025842A1 (en) * 2000-08-31 2002-02-28 Konami Corporation Game machine, game processing method and information storage medium
US20070234885A1 (en) * 2006-03-29 2007-10-11 Harmonix Music Systems, Inc. Game controller simulating a musical instrument

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1029565A2 (en) * 1999-02-16 2000-08-23 Konami Co., Ltd. Music staging game apparatus, music staging game method, and readable storage medium
US20020019258A1 (en) * 2000-05-31 2002-02-14 Kim Gerard Jounghyun Methods and apparatus of displaying and evaluating motion data in a motion game apparatus
US20020025842A1 (en) * 2000-08-31 2002-02-28 Konami Corporation Game machine, game processing method and information storage medium
US20070234885A1 (en) * 2006-03-29 2007-10-11 Harmonix Music Systems, Inc. Game controller simulating a musical instrument

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
YANG U. ET AL: "Implementation and Evaluation of "Just Follow Me": An Immersive, VR-based, Motion-Training System", PRESENCE: TELEOPERATORS AND VIRTUAL ENVIRONMENTS, vol. 11, no. 3, June 2002 (2002-06-01), MIT PRESS CAMBRIDGE, MA, USA, pages 304 - 323, Retrieved from the Internet <URL:https://svn.v2.nl/andres/Documentation/AR20Behaviour/VR_motion_training.pdf> [retrieved on 20110502] *

Cited By (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8686269B2 (en) 2006-03-29 2014-04-01 Harmonix Music Systems, Inc. Providing realistic interaction to a player of a music-based video game
US8678896B2 (en) 2007-06-14 2014-03-25 Harmonix Music Systems, Inc. Systems and methods for asynchronous band interaction in a rhythm action game
US8444486B2 (en) 2007-06-14 2013-05-21 Harmonix Music Systems, Inc. Systems and methods for indicating input actions in a rhythm-action game
US8439733B2 (en) 2007-06-14 2013-05-14 Harmonix Music Systems, Inc. Systems and methods for reinstating a player within a rhythm-action game
US8678895B2 (en) 2007-06-14 2014-03-25 Harmonix Music Systems, Inc. Systems and methods for online band matching in a rhythm action game
US8663013B2 (en) 2008-07-08 2014-03-04 Harmonix Music Systems, Inc. Systems and methods for simulating a rock band experience
US8449360B2 (en) 2009-05-29 2013-05-28 Harmonix Music Systems, Inc. Displaying song lyrics and vocal cues
US8465366B2 (en) 2009-05-29 2013-06-18 Harmonix Music Systems, Inc. Biasing a musical performance input to a part
US10421013B2 (en) 2009-10-27 2019-09-24 Harmonix Music Systems, Inc. Gesture-based user interface
US10357714B2 (en) 2009-10-27 2019-07-23 Harmonix Music Systems, Inc. Gesture-based user interface for navigating a menu
US9981193B2 (en) 2009-10-27 2018-05-29 Harmonix Music Systems, Inc. Movement based recognition and evaluation
US8636572B2 (en) 2010-03-16 2014-01-28 Harmonix Music Systems, Inc. Simulating musical instruments
US8550908B2 (en) 2010-03-16 2013-10-08 Harmonix Music Systems, Inc. Simulating musical instruments
US8874243B2 (en) 2010-03-16 2014-10-28 Harmonix Music Systems, Inc. Simulating musical instruments
US8568234B2 (en) 2010-03-16 2013-10-29 Harmonix Music Systems, Inc. Simulating musical instruments
US9278286B2 (en) 2010-03-16 2016-03-08 Harmonix Music Systems, Inc. Simulating musical instruments
US8702485B2 (en) 2010-06-11 2014-04-22 Harmonix Music Systems, Inc. Dance game and tutorial
US9358456B1 (en) 2010-06-11 2016-06-07 Harmonix Music Systems, Inc. Dance competition game
US8444464B2 (en) 2010-06-11 2013-05-21 Harmonix Music Systems, Inc. Prompting a player of a dance game
US8562403B2 (en) 2010-06-11 2013-10-22 Harmonix Music Systems, Inc. Prompting a player of a dance game
US9024166B2 (en) 2010-09-09 2015-05-05 Harmonix Music Systems, Inc. Preventing subtractive track separation
US10885333B2 (en) 2012-09-12 2021-01-05 2Mee Ltd Augmented reality apparatus and method
US11361542B2 (en) 2012-09-12 2022-06-14 2Mee Ltd Augmented reality apparatus and method
GB2524402B (en) * 2014-03-20 2017-11-22 2Mee Ltd Augmented Reality Apparatus and Method
US10856037B2 (en) 2014-03-20 2020-12-01 2MEE Ltd. Augmented reality apparatus and method
GB2524402A (en) * 2014-03-20 2015-09-23 2Mee Ltd Augmented Reality Apparatus and Method
US11363325B2 (en) 2014-03-20 2022-06-14 2Mee Ltd Augmented reality apparatus and method
US10679413B2 (en) 2014-06-10 2020-06-09 2Mee Ltd Augmented reality apparatus and method
US11094131B2 (en) 2014-06-10 2021-08-17 2Mee Ltd Augmented reality apparatus and method
WO2018198909A1 (en) * 2017-04-28 2018-11-01 株式会社ソニー・インタラクティブエンタテインメント Information processing device, information processing method, and program
US11112857B2 (en) 2017-04-28 2021-09-07 Sony Interactive Entertainment Inc. Information processing apparatus, information processing method, and program
CN109248414A (en) * 2018-09-30 2019-01-22 深圳市科迈爱康科技有限公司 Training based reminding method, device, equipment and readable storage medium storing program for executing
CN109248414B (en) * 2018-09-30 2021-08-10 深圳市科迈爱康科技有限公司 Exercise training reminding method, device and equipment and readable storage medium

Also Published As

Publication number Publication date
FI20096276A0 (en) 2009-12-02

Similar Documents

Publication Publication Date Title
WO2011067469A1 (en) A method, a system and a computer program for instructing and evaluating full-body movement
US10821347B2 (en) Virtual reality sports training systems and methods
US8202161B2 (en) Virtual game instructor
KR101334637B1 (en) Simulating performance of virtual camera
CN102413886B (en) Show body position
US11826628B2 (en) Virtual reality sports training systems and methods
US20100162177A1 (en) Interactive entertainment system and method of operation thereof
US20110293144A1 (en) Method and System for Rendering an Entertainment Animation
US20150182855A1 (en) Motion detection for existing portable devices
JP2013248108A (en) Game system, game processing method, game device and game program
EP2969078B1 (en) User-generated recordings of skeletal animations
US10885691B1 (en) Multiple character motion capture
WO2016036425A1 (en) Motion detection for portable devices
JP2013172962A (en) Method and/or system for controlling virtual object
KR20200145698A (en) Method and terminal unit for providing 3d assembling puzzle based on augmented reality
JP2010187940A (en) Game device, game control method and program
US9751019B2 (en) Input methods and devices for music-based video games
KR20200076373A (en) The system and method of recognizing a motion
JP2018185824A (en) Application control program, application control method, and application control system
JP4695919B2 (en) GAME PROGRAM AND GAME DEVICE
JP6600051B2 (en) Application control program, application control method, and application control system
Paelke et al. Vision-based interaction-a first glance at playing MR games in the real-world around us
Davidson An evaluation of visual gesture based controls for exploring three dimensional environments
JP7457753B2 (en) PROGRAM AND INFORMATION PROCESSING APPARATUS
JP5133380B2 (en) GAME DEVICE, GAME DEVICE CONTROL METHOD, AND PROGRAM

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10834280

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 10834280

Country of ref document: EP

Kind code of ref document: A1