WO2015073083A1 - Systems and methods for immersive backgrounds - Google Patents

Systems and methods for immersive backgrounds Download PDF

Info

Publication number
WO2015073083A1
WO2015073083A1 PCT/US2014/051671 US2014051671W WO2015073083A1 WO 2015073083 A1 WO2015073083 A1 WO 2015073083A1 US 2014051671 W US2014051671 W US 2014051671W WO 2015073083 A1 WO2015073083 A1 WO 2015073083A1
Authority
WO
WIPO (PCT)
Prior art keywords
displays
platform
user
play area
immersive
Prior art date
Application number
PCT/US2014/051671
Other languages
French (fr)
Inventor
Scott KIER
Matthew KIER
David KIER
Original Assignee
Kier Scott
Kier Matthew
Kier David
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kier Scott, Kier Matthew, Kier David filed Critical Kier Scott
Publication of WO2015073083A1 publication Critical patent/WO2015073083A1/en

Links

Classifications

    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/90Constructional details or arrangements of video game devices not provided for in groups A63F13/20 or A63F13/25, e.g. housing, wiring, connections or cabinets
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20Input arrangements for video game devices
    • A63F13/21Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/213Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells

Definitions

  • Augmented Reality tools and toys based on marker-based object recognition in photos or video stream captures, or using technologies such as Near Field Communication (NFC) or Radio Frequency Identification (RFID).
  • NFC Near Field Communication
  • RFID Radio Frequency Identification
  • Object class recognition e.g., detecting "cars” or “faces”
  • marking "markerless” object recognition is also available.
  • systems and methods are disclosed to create and manage an immersive environment for play that is enhanced with audio and/or video. Both generic themes and images or licensed materials are physically combined and managed by object recognition technologies to create the experience.
  • an immersive interactive system in another aspect, includes a platform with specialized panel(s) supporting placement of one or more static background scenes and one or more displays; a play/engagement area proximal to the platform; an object recognizer to detect one or more items of interest on or near the play/engagement area; one or more sensors to detect one or more user activities on the platform; and a processor coupled to the one or more displays, wherein the processor correlates and serves video and optional audio to the one or more displays based on detected object activities and sensor data.
  • Implementations of the above system can include one or more of the following.
  • the system provides physical mounting options for one or more display monitors and/or printed materials of various sizes combined with independent sensors or those included in mobile devices such as tablet computers or phones. It thus allows the displays of the system to replicate "cut-out" regions of printed or displayed imagery (commonly referred to as regions of interest or "ROI” in the art) to provide interactive and immersive experiences.
  • the software correlates and serves video and optional audio to one or more displays based on object recognition services and sensor inputs.
  • the system includes a shared data store that goes beyond recognizing object classes (e.g., faces, eyes, dolls) to specific items selected by the users (e.g., PlaymobilTM Green Dragon, Red Dragon or more specifically the "Red Dragon with Mounted Knight"), and objects or actors in specific states (e.g., "Red Dragon with Open Mouth” or “Sally Doll with Red Dress”).
  • object classes e.g., faces, eyes, dolls
  • objects or actors in specific states e.g., "Red Dragon with Open Mouth” or “Sally Doll with Red Dress”
  • the software is aware of the user's configuration of display devices(s) and associates specific enhancements to each ROI according to the objects sensed in the play area.
  • the present invention is an system designed to enhance pre-defined image "backgrounds" (immersive landscapes) such as posters or play fields with customized displays that change based on objects recognized by one or more sensors, thereby creating an immersive play environment for physical and imaginative play with real world objects in a manner determined by the actor(s) engaged in play.
  • the system enhances physical play by sensing objects and enhancing an environment created with static images in vertical or horizontal mountings coupled with one or more sensing and display devices and CPUs running the software.
  • the system is designed to complement and enhance the play experience with existing objects, not to replace tactile, cooperative or competitive imaginative engagements such as play.
  • FIG. 1A is an illustration of an example set up process that is operable to provide immersive theme based content interaction techniques.
  • FIG. IB shows an exemplary physical environment in an example implementation that is operable to provide immersive theme based content interaction techniques.
  • FIG. 1C shows an exemplary top view of the system of FIG. IB.
  • FIG. 2 shows an exemplary software architecture to provide immersive theme based content interaction techniques.
  • FIG. 3 shows an exemplary front view of the system
  • FIG. 4 shows an exemplary side view of the system
  • FIG.5 shows an exemplary rear view of the system.
  • FIGS. 6-7 show an exemplary output selection and customization flow.
  • FIG. 8 shows an exemplary front view of the system with the physical design for a corner
  • FIG. 9 shows a bottom view displaying a horizontally mounted display.
  • a system that is integrated with markerless recognition software and systems geared for play in the physical world (e.g., to enhance the experience of playing with real toys in a markerless and open fashion) is disclosed.
  • the present invention combines unique physical characteristics and software components to create such an environment.
  • FIG. 1A is an illustration of an example set up process that is operable to provide theme based content interaction techniques.
  • the invention provides an immersive play environment, presenting a functional and attractive background and/or platform for "360 degree play" with objects in real space.
  • One or more of the system may be combined to provide further immersion or interaction as well: for example, a play mat of a forest floor with embedded displays and a vertical background poster which also has one or more displays, or vertically mounted displays could be configured at varied depths from or angles to the playfield.
  • the system can be packaged with pre-configured software and/or devices for sensing and display, may be customized by the user (licensee or consumer), or both.
  • the core functionality is enabled through a process performed during manufacturing and/or by the end user.
  • FIG. 1A illustrates the key elements of this configuration process.
  • block 110 involves the selection or creation of a "theme" for the implementation of the system.
  • a theme could be a generic background for play, such as a forest or mountain scene, a theme specific to a consumer product (or product family) or any other image suitable to the end user's vision for a background/stage for play.
  • Block 120 displays the process of customizing printed media and mounting feature(s) of the system to integrate the desired display(s).
  • display devices are mounted flush to a printed poster in a position such that the full or partial display device fills a cut-out section of the poster.
  • Block 130 displays the configuration process for one or more sensors used with the system.
  • Block 140 displays the process of selecting or configuring the "Object Library" to be used for object detection.
  • Packaged data stores for common consumer objects will be available for selection (logically these will be packaged with the images used for theme selection). Additionally, end users will be able to "train" the object recognition software to their own objects through the invention.
  • Objects may be defined as specific actors (i.e., an individual person) or embellishments available for play (e.g., clothing for a doll, a weapon for a toy soldier).
  • Block 150 displays the final configuration decisions for use, which include confirming or selecting the video, images and/or sounds correlated to the objects. Again these will be provided in conjunction with the theme concept, however it is contemplated that end users may want to add or "de-select" specific interactions to meet their needs, and this capability is also provided by the software components of the present invention. This repository is then used by the runtime software to deliver the appropriate experience to the user(s) via one or more display devices.
  • Example procedures are then described which may be employed by the example environment, as well as in other environments.
  • FIG. IB shows an exemplary physical environment in an example implementation that is operable to provide immersive theme based content interaction techniques.
  • the example shows a combination of hardware and software that provides enhanced play for children of any age.
  • the immersive experience is based on real-time interaction with the physical world.
  • the system enhances imaginations and extends play time.
  • a background panel 10 is positioned near a playfield 20.
  • the background panel 10 shows a static image that is supplemented with an active display 12 to emphasize actions that may be present in the game.
  • one or more cameras are configured by the system to monitor the play field 20 and capture images for processing actions.
  • the display 12 is dynamically changed to enhance the static background imagery.
  • the camera capture is used primarily for object or actor identification, not necessarily for redisplay.
  • Image enhancement corresponds to the found objects: in this example, the static mountain is enhanced with a spreading fire and roaring flames associated with a dragon 22 in the foreground.
  • the active display 12 can be an HDTV monitor and is brightened for highlight.
  • a safety-ensured table or platform can include specialized panel(s) supporting the placement of various preconfigured or user-provided displays.
  • an integrated structure supports one or more display devices with hidden power and cable management.
  • a user may select a theme to customize aspects of a user interface used for interactions with a game provider, a service provider, computer, operating system, among others.
  • the play themes are used to customize a set to one or more consumer products, such as toys.
  • the themes can be used by a content provider provide a consistent user experience across interaction that may occur with different hardware and corresponding user interfaces. Techniques are described to provide theme based content interaction.
  • a theme as used herein may be a collection of settings for a user interface that are selected to correspond to a particular topic.
  • themes such as themes related to hobbies, family, social groups, work, entertainment, services, people, products, music, and so forth.
  • a theme may be centered around "Sports" or a favorite professional sports team.
  • Another theme may be centered around a user's favorite celebrity.
  • Yet another theme may be centered around a hobby of a user, such as a "Garden” theme.
  • a variety of other examples of themes are also contemplated.
  • a user interface is provided to enable interaction with a variety of services from a user, a theme developer, or a game or entertainment provider.
  • One or more theme preferences are stored that describe a theme that may be specified by a user through interaction with a game provider. Accordingly, when the user interacts with a particular game provided by the game provider, the game provider may detect a theme associated with the user and output a user interface to include content portions customized to correspond to the detected theme.
  • the theme may be applied consistently across a variety of services and user interfaces that are provided through the service provider.
  • Variable content portions of the user interfaces may be changed to include content that matches a detected theme. Further content portions related to a theme may be added and removed from a user interface based upon a detected theme to provide theme specific functionality. In this way, themes may be defined to customize both visual aspects and content aspects of user interfaces through which a user may interact with a game provider.
  • FIG. 1C shows an exemplary top view of the system of FIG. IB.
  • the system may have a plurality of displays 1-X marked as Blocks 315 and 316.
  • the system may have a plurality of sensors such as Markers 320 and 321 that provides a plurality of fields of view over and around a "playfield" marked as 305.
  • An Object Monitor & Controller (230) is implemented as an application that communicates with one or more inputs as well as one or more displays.
  • the display may be part of the host for the application (as in a smartphone or tablet), or may be external to the device hosting the application.
  • the functions and features are as follows: Configuration Engine Load and store internal and customization data "Plug and Play" feel for easy home setup
  • Custom training and recognition is fast and frames/samples per second and ROI, for accurate
  • Options for content control e.g., age-based,
  • FIG. 2 shows an exemplary software architecture.
  • the architecture uses a central server component 200 to function as the "Product Configuration Engine” over Internet-based or local server networking technologies as displayed at marker 221.
  • the engine 200 communicates with the '"local" software components 230 that drive the user interaction with the system.
  • the Controller 230 may be implemented as a software application that runs locally on user-provided hardware or may be implemented as an "appliance" included with the system.
  • a Configuration Manager 205 manages availability, configuration and distribution of the software by also leveraging commercially available software for network and supporting hardware interfaces displayed as 221. Configurations for user and license management are stored centrally in the data store marked as 210. Configuration details and settings for creating and customizing the user experience are centrally managed and maintained (as themes and customization settings) in data store 212, while required hardware-specific details, including interactions with themes, is managed and maintained in data store 214. Marker 220 represents the communications components for connecting the central server to remote or local installations, using commercially available networking and communications protocols.
  • a local software system 230 replicates the base architecture for the main server but the content of the data stores applies only to an implementation of one or more systems, as defined by the user(s) and accounts maintained in components 210 and 250.
  • Configurations (initial and ongoing maintenance) are synchronized via the link(s) between 220 and 222, the server-side and local Communications Managers.
  • the local Configuration Manager 240 could be updated using information from local or external storage devices as shown by 245, such as USB storage or other similar devices.
  • the "local" data stores 250, 252 and 254 are technically analogous to components 210, 212 and 214 however only contain data relevant to the defined installation/implementation.
  • a Runtime Controller 260 provides the software dedicated to providing the user experience. Sensor inputs and object recognition are leveraged and managed according the defined configuration(s) of hardware and selected interactions/themes, and are then produced by sending data, signals or other information to be output via the Input/Output (I/O) controller 270, which provides connections to the physical sensors 275 and output devices 280. I/O Controller may also communicate with adapters and devices via wired or wireless technologies as showed in the optional Block 285.
  • FIGS. 3-5 shows a schematic diagram of one exemplary physical implementation of the system. Turning now to FIG. 3, an exemplary front view of the system is shown.
  • the base frame of the system is marked as 300 and represents a table-like base connected to an upright structure 310, both of which can support the above referenced displays, with examples shown as 315 and 316 in the vertical position.
  • the vertical structure 310 supports configurable hardware for mounting the displays such that their surfaces are flush to the mounted poster marked as 311.
  • the horizontal structure 305 can support either a tabletop or mounting hardware similar to the vertical frame.
  • Markers 320 and 321 show an in- device sensor and frame-mounted sensor, respectively. From the frontal view, the front edge of the raised shelf-like walkway 325 between the tabletop surface and vertical frame is visible.
  • the optional walkway can provide the "360 degree" access to the horizontal platform and assist with overall stability as well as cable management.
  • FIG. 4 shows an exemplary side view of the system.
  • Marker 410 displays the vertical frame that includes the mounting hardware for the printed image 311 and the exemplary displays 315 and 316.
  • the mounting hardware allows for any logical configuration, and could also be implemented in the horizontal base frame 305.
  • the horizontal frame is supported by legs 430 and 431 attached to the frame with optional hinges 435 and 436 to provide for portability and easier storage.
  • Marker 325 shows the optional raised walkway that can provide separation between the horizontal and vertical frames and enable hidden cable management.
  • FIG.5 shows an exemplary rear view of the system.
  • the frame 300 is shown with attached mounting hardware 540.
  • the mounting hardware is both removable and configurable using sliding and bracing attachments, with examples at 545A and 545B, and allows the attachment of displays 315 and 316 in the manner described above as well as sensors such as 321 and the poster image 311 in the exemplary system.
  • the mounting hardware 540, 545A and 545 also display that the structure provides cable hiding/management features to provide a "clean" and more immersive experience to the end users.
  • construction options for the playfield 20 may include one or more of the following:
  • Playset-specific "Playmat” type covers for table surfaces (Playfield)
  • Proprietary/licensed imagery that is used in tandem with custom "scenes” or animations (with or without sound) to create a more immersive environment
  • FIG. 6 shows an exemplary output selection and customization flow.
  • FIG. 6 displays the execution process of the software in run time, following the configuration process of FIG. 1A, completed with Marker 150. The details of the process begin with 1510, starting of the software services for sensing and display control.
  • 1520 displays the startup phase, which verifies and selects the appropriate software services to be run in memory attached to or supporting one or more CPUs.
  • Sensor configuration 1522 relies on a data store 1523 of all selected/available sensors, and "activates" the sensors for operation during runtime by validating their active connections and settings (for example, a frame sampling rate for a camera sensor).
  • 1524 validates configured displays and their configurations to select and load to memory the proper software components and driving settings for the display outputs based on the data store 1525 (e.g., display size and resolution to a large device connected via HDMI cable versus a smaller size tablet connected over a wireless network via a dongle).
  • the data store 1525 e.g., display size and resolution to a large device connected via HDMI cable versus a smaller size tablet connected over a wireless network via a dongle.
  • Object sensing is configured to specific objects, products or classes in Step 140, and the results provide the inputs to 1526 via the data store 1527, including images and vector data required for markerless sensing commonly known as "positive and negative views" in the art.
  • the final key step in the instantiation of the software is the loading and preparation of the keys to the resulting actions from object detection.
  • Each target object will have one or more audio and/or video options, and these are managed via the process 1528 based on the data store 1529, which contains both links to the series of outputs and the files or data streams themselves.
  • Module 1530 displays "Sensor Managers" - one or more processes running on the CPU according to the processes of 1520, with one per sensor or sensor configuration (e.g., one process may manage object detection on a camera sensor, and another may manage motion detection from the same sensor feed).
  • Each Sensor Manager leverages information in memory or in the data store 1527 and is able to pass trigger events to 1540, which associate actions to the events based on the information loaded in setup process 1528, or in real-time interaction with the data store 1529.
  • the process marked 1550 manages the ongoing execution of events, including the monitoring of ongoing actions, such as an audio or video stream currently in play.
  • the process manages queues and streams based on attributes passed from the prior processes and current state data.
  • Marker 1551 illustrates that specific actions may also send triggers or requests to the same process via manual selection or input, such as on a control User Interface.
  • Process 1560 uses information in memory or the data store 1525 to function as the Output Controller and executes the instructions passed from 1550 to create the user's experience in real time.
  • FIG. 7 further expands on the processing engine described in FIG. 6 and shows the operation of marker 1550 and its interaction with related components of the software.
  • 1600 represents the in- memory variables (global and instance data for each process) that are available to the overall decision processing instance 1550 and decision engines 1620 and 1625.
  • Process 1540 detects an object from a sensor, and passes a trigger to the main process, which determines if a prior detection 1610 has resulted in a current display-in-progress 1612.
  • this process may consider "in progress" to include any lag time settings or other configurations that are designed to manage the user experience.
  • These variables lead to the determination of process 1614 of a "display complete" (including audio) decision that would allow an action to be executed.
  • Marker 1615 receives a request for action and in conjunction with the rules processor 1620, accesses memory and/or the data store 1528 to determine the next appropriate action for the detection event (e.g., "play the third audio/video (A/V) stream configured for Object X, as the prior one displayed for that object 4 minutes ago was A/V Stream 2").
  • This process also includes access to in-memory variables 1600 and saved logs 1700.
  • the rules engine 1620 then passes instructions to a Display Controller process 1625, which leverages the in- memory information and data store 1524 to prepare and send execution instructions to the Output Controller 1560.
  • FIGS. 8&9 display front and bottom view of a second exemplary model. These examples display the horizontal platform 805 configured with multiple vertical displays 810 and 811. The physical structures support vertical displays 815 and 816, in addition to a horizontal display mounted in the horizontal platform/table. This exemplary instance also displays a frame-mounted optical camera sensor at marker 825.
  • a theme may be applied to customize content aspects of user interfaces output locally at a computer, such as applying the theme to customize content portions of an operating system user interface of a computer, a desktop gadget output on a computer desktop, a user interface of a desktop application, and so forth.
  • a variety of other examples are also contemplated.
  • the invention may be implemented in hardware, firmware or software, or a combination of the three.
  • the invention is implemented in a computer program executed on a programmable computer having a processor, a data storage system, volatile and non-volatile memory and/or storage elements, at least one input device and at least one output device.
  • the computer preferably includes a processor, random access memory (RAM), a program memory
  • I/O controller is coupled by means of an I/O bus to an I/O interface.
  • I/O interface receives and transmits data in analog or digital form over communication links such as a serial link, local area network, wireless link, and parallel link.
  • a display, a keyboard and a pointing device may also be connected to I/O bus.
  • separate connections may be used for I/O interface, display, keyboard and pointing device.
  • Programmable processing system may be
  • preprogrammed or it may be programmed (and reprogrammed) by downloading a program from another source (e.g., a floppy disk, CD-ROM, or another computer).
  • another source e.g., a floppy disk, CD-ROM, or another computer.
  • Each computer program is tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein.
  • the inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.
  • first and second are used herein to describe various features, elements, regions, layers and/or sections, these features, elements, regions, layers and/or sections should not be limited by these terms. These terms are only used to distinguish one feature, element, region, layer or section from another feature, element, region, layer or section. Thus, a first feature, element, region, layer or section discussed below could be termed a second feature, element, region, layer or section, and similarly, a second without departing from the teachings of the present invention.
  • Coupled to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present.

Abstract

An immersive interactive system includes a platform with specialized panel(s) supporting placement of a static background scene and one or more displays; a play area proximal to the platform; an object recognizer to detect one or more items of interest on or near the play area; one or more sensors to detect one or more user activities on the platform; and a processor coupled to the one or more displays, wherein the processor correlates and serves video and optional audio to the one or more displays based on detected object activities and sensor data.

Description

SYSTEMS AND METHODS FOR IMMERSIVE BACKGROUNDS
BACKGROUND
Many products and inventions have been developed in recent years using "marker-based" technologies for object recognition or other technologies to create entertainment systems under the umbrella of augmented reality or to enhance a video gaming experience. Primarily, these inventions morph or superimpose graphics on an entity or physical object captured via an electro-optical camera collection and then "virtualize" the combined images.
Various consumer and commercial products and technologies are available for Augmented Reality tools and toys based on marker-based object recognition in photos or video stream captures, or using technologies such as Near Field Communication (NFC) or Radio Frequency Identification (RFID). Object class recognition (e.g., detecting "cars" or "faces") with "markerless" object recognition is also available.
US Patents 8,316,298, 8,542,250 and 8,549,440 as well as offerings from video game makers provide marker-based or motion-based technologies. However conventional systems focus on extending a video game or video stream from and on the sensing device itself, such as a handheld gaming device, tablet or phone, or enhancing the imagery in a video game display.
SUMMARY
In one aspect, systems and methods are disclosed to create and manage an immersive environment for play that is enhanced with audio and/or video. Both generic themes and images or licensed materials are physically combined and managed by object recognition technologies to create the experience.
In another aspect, an immersive interactive system is disclosed that includes a platform with specialized panel(s) supporting placement of one or more static background scenes and one or more displays; a play/engagement area proximal to the platform; an object recognizer to detect one or more items of interest on or near the play/engagement area; one or more sensors to detect one or more user activities on the platform; and a processor coupled to the one or more displays, wherein the processor correlates and serves video and optional audio to the one or more displays based on detected object activities and sensor data.
Implementations of the above system can include one or more of the following. The system provides physical mounting options for one or more display monitors and/or printed materials of various sizes combined with independent sensors or those included in mobile devices such as tablet computers or phones. It thus allows the displays of the system to replicate "cut-out" regions of printed or displayed imagery (commonly referred to as regions of interest or "ROI" in the art) to provide interactive and immersive experiences. The software correlates and serves video and optional audio to one or more displays based on object recognition services and sensor inputs. The system includes a shared data store that goes beyond recognizing object classes (e.g., faces, eyes, dolls) to specific items selected by the users (e.g., Playmobil™ Green Dragon, Red Dragon or more specifically the "Red Dragon with Mounted Knight"), and objects or actors in specific states (e.g., "Red Dragon with Open Mouth" or "Sally Doll with Red Dress"). The software is aware of the user's configuration of display devices(s) and associates specific enhancements to each ROI according to the objects sensed in the play area.
Advantages of the system may include one or more of the following. The present invention is an system designed to enhance pre-defined image "backgrounds" (immersive landscapes) such as posters or play fields with customized displays that change based on objects recognized by one or more sensors, thereby creating an immersive play environment for physical and imaginative play with real world objects in a manner determined by the actor(s) engaged in play. The system enhances physical play by sensing objects and enhancing an environment created with static images in vertical or horizontal mountings coupled with one or more sensing and display devices and CPUs running the software. The system is designed to complement and enhance the play experience with existing objects, not to replace tactile, cooperative or competitive imaginative engagements such as play.
BRIEF DESCRIPTION OF THE DRAWINGS
The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the FIG. in which the reference number first appears. The use of the same reference numbers in different instances in the description and the figures may indicate similar or identical items.
FIG. 1A is an illustration of an example set up process that is operable to provide immersive theme based content interaction techniques.
FIG. IB shows an exemplary physical environment in an example implementation that is operable to provide immersive theme based content interaction techniques.
FIG. 1C shows an exemplary top view of the system of FIG. IB.
FIG. 2 shows an exemplary software architecture to provide immersive theme based content interaction techniques.
FIG. 3 shows an exemplary front view of the system; FIG. 4 shows an exemplary side view of the system, while FIG.5 shows an exemplary rear view of the system.
FIGS. 6-7 show an exemplary output selection and customization flow.
FIG. 8 shows an exemplary front view of the system with the physical design for a corner, while FIG. 9 shows a bottom view displaying a horizontally mounted display. DESCRIPTION
A system that is integrated with markerless recognition software and systems geared for play in the physical world (e.g., to enhance the experience of playing with real toys in a markerless and open fashion) is disclosed. The present invention combines unique physical characteristics and software components to create such an environment.
FIG. 1A is an illustration of an example set up process that is operable to provide theme based content interaction techniques. Once setup/configuration is complete, the invention provides an immersive play environment, presenting a functional and attractive background and/or platform for "360 degree play" with objects in real space. One or more of the system may be combined to provide further immersion or interaction as well: for example, a play mat of a forest floor with embedded displays and a vertical background poster which also has one or more displays, or vertically mounted displays could be configured at varied depths from or angles to the playfield. The system can be packaged with pre-configured software and/or devices for sensing and display, may be customized by the user (licensee or consumer), or both. The core functionality is enabled through a process performed during manufacturing and/or by the end user. FIG. 1A illustrates the key elements of this configuration process.
Turning now to FIG. 1A, block 110 involves the selection or creation of a "theme" for the implementation of the system. A theme could be a generic background for play, such as a forest or mountain scene, a theme specific to a consumer product (or product family) or any other image suitable to the end user's vision for a background/stage for play. Block 120 displays the process of customizing printed media and mounting feature(s) of the system to integrate the desired display(s). In the exemplary implementation, display devices are mounted flush to a printed poster in a position such that the full or partial display device fills a cut-out section of the poster. Block 130 displays the configuration process for one or more sensors used with the system. Many sensor options could be leveraged, such as integrated sensors in a display device (e.g., a tablet or other personal computing device or phone), various single- and multiple-point sensing optical cameras or light-cameras (plenoptic cameras) and other sensors such as infrared (IR) that sense parts of the spectrum outside of human vision capabilities. Configuration may simply include mounting or selection from a list presented by the included software, or may require more technical configuration for calibration of the sensor(s) and software. Block 140 displays the process of selecting or configuring the "Object Library" to be used for object detection. Packaged data stores for common consumer objects will be available for selection (logically these will be packaged with the images used for theme selection). Additionally, end users will be able to "train" the object recognition software to their own objects through the invention. Within this training process, Objects may be defined as specific actors (i.e., an individual person) or embellishments available for play (e.g., clothing for a doll, a weapon for a toy soldier).
Block 150 displays the final configuration decisions for use, which include confirming or selecting the video, images and/or sounds correlated to the objects. Again these will be provided in conjunction with the theme concept, however it is contemplated that end users may want to add or "de-select" specific interactions to meet their needs, and this capability is also provided by the software components of the present invention. This repository is then used by the runtime software to deliver the appropriate experience to the user(s) via one or more display devices.
In the following discussion, an example environment is first described that is operable to employ theme based immersive content interaction techniques. Example procedures are then described which may be employed by the example environment, as well as in other environments.
FIG. IB shows an exemplary physical environment in an example implementation that is operable to provide immersive theme based content interaction techniques. The example shows a combination of hardware and software that provides enhanced play for children of any age. The immersive experience is based on real-time interaction with the physical world. The system enhances imaginations and extends play time. A background panel 10 is positioned near a playfield 20. The background panel 10 shows a static image that is supplemented with an active display 12 to emphasize actions that may be present in the game. During play, one or more cameras are configured by the system to monitor the play field 20 and capture images for processing actions. Unlike traditional Augmented Reality systems, the display 12 is dynamically changed to enhance the static background imagery. The camera capture is used primarily for object or actor identification, not necessarily for redisplay. Image enhancement corresponds to the found objects: in this example, the static mountain is enhanced with a spreading fire and roaring flames associated with a dragon 22 in the foreground.
As shown in the embodiment, the active display 12 can be an HDTV monitor and is brightened for highlight. A safety-ensured table or platform can include specialized panel(s) supporting the placement of various preconfigured or user-provided displays. Further, an integrated structure supports one or more display devices with hidden power and cable management. A user may select a theme to customize aspects of a user interface used for interactions with a game provider, a service provider, computer, operating system, among others. The play themes are used to customize a set to one or more consumer products, such as toys. Further, the themes can be used by a content provider provide a consistent user experience across interaction that may occur with different hardware and corresponding user interfaces. Techniques are described to provide theme based content interaction. A theme as used herein may be a collection of settings for a user interface that are selected to correspond to a particular topic. A variety of different themes are contemplated, such as themes related to hobbies, family, social groups, work, entertainment, services, people, products, music, and so forth. For example, a theme may be centered around "Sports" or a favorite professional sports team. Another theme may be centered around a user's favorite celebrity. Yet another theme may be centered around a hobby of a user, such as a "Garden" theme. A variety of other examples of themes are also contemplated.
In an implementation, a user interface is provided to enable interaction with a variety of services from a user, a theme developer, or a game or entertainment provider. One or more theme preferences are stored that describe a theme that may be specified by a user through interaction with a game provider. Accordingly, when the user interacts with a particular game provided by the game provider, the game provider may detect a theme associated with the user and output a user interface to include content portions customized to correspond to the detected theme. The theme may be applied consistently across a variety of services and user interfaces that are provided through the service provider. Variable content portions of the user interfaces may be changed to include content that matches a detected theme. Further content portions related to a theme may be added and removed from a user interface based upon a detected theme to provide theme specific functionality. In this way, themes may be defined to customize both visual aspects and content aspects of user interfaces through which a user may interact with a game provider.
FIG. 1C shows an exemplary top view of the system of FIG. IB. The system may have a plurality of displays 1-X marked as Blocks 315 and 316. The system may have a plurality of sensors such as Markers 320 and 321 that provides a plurality of fields of view over and around a "playfield" marked as 305. An Object Monitor & Controller (230) is implemented as an application that communicates with one or more inputs as well as one or more displays. The display may be part of the host for the application (as in a smartphone or tablet), or may be external to the device hosting the application. In one exemplary objection monitor and controller, the functions and features are as follows: Configuration Engine Load and store internal and customization data "Plug and Play" feel for easy home setup
Hardware recognition and self-configuration Browser-based setup completion
Communication with servers/hosts for updates Automatic updates from server (if Internet and licensing connection present) or via USB storage
Privacy and DRM (digital rights management) Data collection arid reporting within secure & private layers
Computer Ability to use pre-loaded libraries as well as Packages will be available "off the shelf" for Vision/Object Recognition "train" to learn new objects major licenses (e.g., "Malibu Barbie" or
Manager "Chuggington")
Highly self-configurable based on engine
settings and real time conditions: Custom training and recognition is fast and frames/samples per second and ROI, for accurate
example, based on device/sensor
Computer Vision software continues to evolve in the Open Source community
Action Manager Based on settings from the Engine settings and Simple updates/config through XML or other the triggers from the Recognition Manager, this package deliveries from servers or storage engine selects and executes the presentation of media
our immersive features
Options for content control (e.g., age-based,
Manage parallel/multiple detections, existing login/parental restrictions)
presentations and overall user experience
based on configurations that use internal Ability to recognize presentation needs based attributes such as priority, sequence and time on hardware config (e.g., display size,
position, resolution)
Populate database and logs of usage statistics
FIG. 2 shows an exemplary software architecture. The architecture uses a central server component 200 to function as the "Product Configuration Engine" over Internet-based or local server networking technologies as displayed at marker 221. The engine 200 communicates with the '"local" software components 230 that drive the user interaction with the system. The Controller 230 may be implemented as a software application that runs locally on user-provided hardware or may be implemented as an "appliance" included with the system.
A Configuration Manager 205 manages availability, configuration and distribution of the software by also leveraging commercially available software for network and supporting hardware interfaces displayed as 221. Configurations for user and license management are stored centrally in the data store marked as 210. Configuration details and settings for creating and customizing the user experience are centrally managed and maintained (as themes and customization settings) in data store 212, while required hardware-specific details, including interactions with themes, is managed and maintained in data store 214. Marker 220 represents the communications components for connecting the central server to remote or local installations, using commercially available networking and communications protocols.
A local software system 230 replicates the base architecture for the main server but the content of the data stores applies only to an implementation of one or more systems, as defined by the user(s) and accounts maintained in components 210 and 250. Configurations (initial and ongoing maintenance) are synchronized via the link(s) between 220 and 222, the server-side and local Communications Managers. Alternatively, the local Configuration Manager 240 could be updated using information from local or external storage devices as shown by 245, such as USB storage or other similar devices. The "local" data stores 250, 252 and 254 are technically analogous to components 210, 212 and 214 however only contain data relevant to the defined installation/implementation.
A Runtime Controller 260 provides the software dedicated to providing the user experience. Sensor inputs and object recognition are leveraged and managed according the defined configuration(s) of hardware and selected interactions/themes, and are then produced by sending data, signals or other information to be output via the Input/Output (I/O) controller 270, which provides connections to the physical sensors 275 and output devices 280. I/O Controller may also communicate with adapters and devices via wired or wireless technologies as showed in the optional Block 285. FIGS. 3-5 shows a schematic diagram of one exemplary physical implementation of the system. Turning now to FIG. 3, an exemplary front view of the system is shown. The base frame of the system is marked as 300 and represents a table-like base connected to an upright structure 310, both of which can support the above referenced displays, with examples shown as 315 and 316 in the vertical position. The vertical structure 310 supports configurable hardware for mounting the displays such that their surfaces are flush to the mounted poster marked as 311. The horizontal structure 305 can support either a tabletop or mounting hardware similar to the vertical frame. Markers 320 and 321 show an in- device sensor and frame-mounted sensor, respectively. From the frontal view, the front edge of the raised shelf-like walkway 325 between the tabletop surface and vertical frame is visible. The optional walkway can provide the "360 degree" access to the horizontal platform and assist with overall stability as well as cable management.
FIG. 4 shows an exemplary side view of the system. Marker 410 displays the vertical frame that includes the mounting hardware for the printed image 311 and the exemplary displays 315 and 316. The mounting hardware allows for any logical configuration, and could also be implemented in the horizontal base frame 305. The horizontal frame is supported by legs 430 and 431 attached to the frame with optional hinges 435 and 436 to provide for portability and easier storage. Marker 325 shows the optional raised walkway that can provide separation between the horizontal and vertical frames and enable hidden cable management.
FIG.5 shows an exemplary rear view of the system. In the rear view diagram, the frame 300 is shown with attached mounting hardware 540. The mounting hardware is both removable and configurable using sliding and bracing attachments, with examples at 545A and 545B, and allows the attachment of displays 315 and 316 in the manner described above as well as sensors such as 321 and the poster image 311 in the exemplary system. The mounting hardware 540, 545A and 545 also display that the structure provides cable hiding/management features to provide a "clean" and more immersive experience to the end users.
Other construction options for the playfield 20 may include one or more of the following:
Folding table and legs for ease of storage; adjustable table height and image/display distance from table edge
Choice of materials for reduced cost and weight/portability Playset-specific "Playmat" type covers for table surfaces ("Playfield")
Additional vertical panels for "wrapping around" the play platform
Proprietary/licensed imagery that is used in tandem with custom "scenes" or animations (with or without sound) to create a more immersive environment
FIG. 6 shows an exemplary output selection and customization flow. FIG. 6 displays the execution process of the software in run time, following the configuration process of FIG. 1A, completed with Marker 150. The details of the process begin with 1510, starting of the software services for sensing and display control. 1520 displays the startup phase, which verifies and selects the appropriate software services to be run in memory attached to or supporting one or more CPUs. Sensor configuration 1522 relies on a data store 1523 of all selected/available sensors, and "activates" the sensors for operation during runtime by validating their active connections and settings (for example, a frame sampling rate for a camera sensor). Similarly, 1524 validates configured displays and their configurations to select and load to memory the proper software components and driving settings for the display outputs based on the data store 1525 (e.g., display size and resolution to a large device connected via HDMI cable versus a smaller size tablet connected over a wireless network via a dongle).
Object sensing is configured to specific objects, products or classes in Step 140, and the results provide the inputs to 1526 via the data store 1527, including images and vector data required for markerless sensing commonly known as "positive and negative views" in the art. The final key step in the instantiation of the software is the loading and preparation of the keys to the resulting actions from object detection. Each target object will have one or more audio and/or video options, and these are managed via the process 1528 based on the data store 1529, which contains both links to the series of outputs and the files or data streams themselves.
Module 1530 displays "Sensor Managers" - one or more processes running on the CPU according to the processes of 1520, with one per sensor or sensor configuration (e.g., one process may manage object detection on a camera sensor, and another may manage motion detection from the same sensor feed). Each Sensor Manager leverages information in memory or in the data store 1527 and is able to pass trigger events to 1540, which associate actions to the events based on the information loaded in setup process 1528, or in real-time interaction with the data store 1529. The process marked 1550 manages the ongoing execution of events, including the monitoring of ongoing actions, such as an audio or video stream currently in play. The process manages queues and streams based on attributes passed from the prior processes and current state data. Marker 1551 illustrates that specific actions may also send triggers or requests to the same process via manual selection or input, such as on a control User Interface. Process 1560 uses information in memory or the data store 1525 to function as the Output Controller and executes the instructions passed from 1550 to create the user's experience in real time.
FIG. 7 further expands on the processing engine described in FIG. 6 and shows the operation of marker 1550 and its interaction with related components of the software. 1600 represents the in- memory variables (global and instance data for each process) that are available to the overall decision processing instance 1550 and decision engines 1620 and 1625. In this processing example, Process 1540 detects an object from a sensor, and passes a trigger to the main process, which determines if a prior detection 1610 has resulted in a current display-in-progress 1612. In addition, this process may consider "in progress" to include any lag time settings or other configurations that are designed to manage the user experience. These variables lead to the determination of process 1614 of a "display complete" (including audio) decision that would allow an action to be executed. Marker 1615 receives a request for action and in conjunction with the rules processor 1620, accesses memory and/or the data store 1528 to determine the next appropriate action for the detection event (e.g., "play the third audio/video (A/V) stream configured for Object X, as the prior one displayed for that object 4 minutes ago was A/V Stream 2"). This process also includes access to in-memory variables 1600 and saved logs 1700. The rules engine 1620 then passes instructions to a Display Controller process 1625, which leverages the in- memory information and data store 1524 to prepare and send execution instructions to the Output Controller 1560.
FIGS. 8&9 display front and bottom view of a second exemplary model. These examples display the horizontal platform 805 configured with multiple vertical displays 810 and 811. The physical structures support vertical displays 815 and 816, in addition to a horizontal display mounted in the horizontal platform/table. This exemplary instance also displays a frame-mounted optical camera sensor at marker 825.
While the exemplary model is contemplated for children's toys primarily, the technology platform and physical characteristics of the invention could be transferrable to marketing platforms, educational play or experiences, and other interactive products. Additionally, while aspects of theme based content interaction techniques are described herein in relation to services and user interfaces provided by a service provider, it is contemplated that the techniques may be employed to extend themes to content aspects of user interfaces output in a variety of settings. For example, a theme may be applied to customize content aspects of user interfaces output locally at a computer, such as applying the theme to customize content portions of an operating system user interface of a computer, a desktop gadget output on a computer desktop, a user interface of a desktop application, and so forth. A variety of other examples are also contemplated.
Although the theme based content interaction techniques have been described in language specific to structural features and/or methodological acts, it is to be understood that the appended claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the theme based content interaction techniques.
The invention may be implemented in hardware, firmware or software, or a combination of the three. Preferably the invention is implemented in a computer program executed on a programmable computer having a processor, a data storage system, volatile and non-volatile memory and/or storage elements, at least one input device and at least one output device.
By way of example, a block diagram of a computer to support the system is discussed next. The computer preferably includes a processor, random access memory (RAM), a program memory
(preferably a writable read-only memory (ROM) such as a flash ROM) and an input/output (I/O) controller coupled by a CPU bus. The computer may optionally include a hard drive controller which is coupled to a hard disk and CPU bus. Hard disk may be used for storing application programs, such as the present invention, and data. Alternatively, application programs may be stored in RAM or ROM. I/O controller is coupled by means of an I/O bus to an I/O interface. I/O interface receives and transmits data in analog or digital form over communication links such as a serial link, local area network, wireless link, and parallel link. Optionally, a display, a keyboard and a pointing device (mouse) may also be connected to I/O bus. Alternatively, separate connections (separate buses) may be used for I/O interface, display, keyboard and pointing device. Programmable processing system may be
preprogrammed or it may be programmed (and reprogrammed) by downloading a program from another source (e.g., a floppy disk, CD-ROM, or another computer).
Each computer program is tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.
Although a few exemplary embodiments of the present invention have been shown and described, the present invention is not limited to the described exemplary embodiments. Instead, it would be appreciated by those skilled in the art that changes may be made to these exemplary embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the description of the embodiments of the invention and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety.
It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that relative terms are intended to encompass different orientations of the device in addition to the orientation depicted in the Figures.
Moreover, it will be understood that although the terms first and second are used herein to describe various features, elements, regions, layers and/or sections, these features, elements, regions, layers and/or sections should not be limited by these terms. These terms are only used to distinguish one feature, element, region, layer or section from another feature, element, region, layer or section. Thus, a first feature, element, region, layer or section discussed below could be termed a second feature, element, region, layer or section, and similarly, a second without departing from the teachings of the present invention.
It will also be understood that when an element is referred to as being "connected" or
"coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements present.
Further, as used herein the term "plurality" refers to at least two elements. Additionally, like numbers refer to like elements throughout.
Thus, there has been shown and described several embodiments of a novel invention. As is evident from the foregoing description, certain aspects of the present invention are not limited by the particular details of the examples illustrated herein, and it is therefore contemplated that other modifications and applications, or equivalents thereof, will occur to those skilled in the art. The terms "having" and "including" and similar terms as used in the foregoing specification are used in the sense of "optional" or "may include" and not as "required".
Many changes, modifications, variations and other uses and applications of the present construction will, however, become apparent to those skilled in the art after considering the specification and the accompanying drawings. All such changes, modifications, variations and other uses and applications which do not depart from the spirit and scope of the invention are deemed to be covered by the invention which is limited only by the claims which follow. The scope of the disclosure is not intended to be limited to the embodiments shown herein, but is to be accorded the full scope consistent with the claims, wherein reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more." All structural and functional equivalents to the elements of the various embodiments described throughout this disclosure that are known or later come to be known to th3ose of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

What is claimed is:
1. (Currently Amended) An immersive interactive system, comprising:
a platform with specialized panel(s) supporting placement of both a static pre-printed background scene and one or more computer controlled displays projecting a motion video on the background scene;
a play area proximal to the platform;
an inanimate object recognizer to detect one or more items of interest on or near the play area wherein the object recognizer identifies one or more features on the inanimate object selected by a user;
one or more sensors to detect the inanimate object on the platform; and a processor coupled to the one or more displays, wherein the processor correlates and serves video and optional audio to the one or more displays based on detected object activities and sensor data, wherein the motion video is customized to the inanimate object.
2. The system of claim 1, wherein the display comprises one or more preconfigured displays or user-provided displays and wherein the one or more displays are dynamically changed to enhance the static background scene.
3. The system of claim 1, wherein the processor displays one or more regions of interest ( OIs) on the one or more displays to provide interactive and immersive experiences.
4. The system of claim 1, wherein the processor receives a user's configuration of the one or more displays and associates predetermined enhancements to each ROI according to one or more objects sensed in the play area.
5. The system of claim 1, wherein an integrated structure to support one or more display devices with hidden power and cable management.
6. The system of claim 1, wherein the background scene is perpendicular to or horizontal with the play area.
7. The system of claim 1, wherein the processor renders licensed imagery used in tandem with custom scenes or animations.
8. The system of claim 1, wherein the platform comprises a foldable table.
9. The system of claim 1, wherein the platform comprises a table with adjustable table height and image or display distance from a table edge.
10. The system of claim 1, wherein each panel provides physical mounting options for one or more display monitors, printed backgrounds or play mats of different sizes.
11. The system of claim 1, wherein the panel provides physical mounting options for one or more sensors.
12. The system of claim 1, comprising one or more vertical panels to wrap around the platform.
13. The system of claim 1, comprising one or more playset-specific mats or covers for a table
surface.
14. The system of claim 1, wherein the platform comprises an integrated structure to support one or more display devices with hidden power and cable management.
15. An immersive interaction method, comprising: setting up a platform with specialized panel(s) supporting placement of a static background scene and one or more displays and a play area proximal to the platform; detecting one or more items of interest on or near the play area; sensing one or more user activities on the platform; and correlating and serving video and optional audio to the one or more displays based on detected object activities and sensor data.
16. The method of claim 15, comprising mounting one or more preconfigured displays or user- provided displays and dynamically updating the one or more displays to enhance the static background scene.
17. The method of claim 15, comprising displaying one or more regions of interest (ROIs) on the one or more displays to provide interactive and immersive experiences.
18. The method of claim 15, comprising associating one or more predetermined enhancements to a region of interest (ROI) according to one or more objects sensed in the play area.
19. The method of claim 15, comprising providing one or more vertical panels to wrap around the platform.
20. The method of claim 15, comprising providing one or more playset-specific mats or covers for a table surface.
PCT/US2014/051671 2013-11-18 2014-08-19 Systems and methods for immersive backgrounds WO2015073083A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US14/082,450 US9011246B1 (en) 2013-11-18 2013-11-18 Systems and methods for immersive backgrounds
US14/082,450 2013-11-18

Publications (1)

Publication Number Publication Date
WO2015073083A1 true WO2015073083A1 (en) 2015-05-21

Family

ID=52822501

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/051671 WO2015073083A1 (en) 2013-11-18 2014-08-19 Systems and methods for immersive backgrounds

Country Status (2)

Country Link
US (1) US9011246B1 (en)
WO (1) WO2015073083A1 (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5832666B2 (en) 2011-12-20 2015-12-16 インテル・コーポレーション Augmented reality representation across multiple devices
US10363486B2 (en) 2013-06-10 2019-07-30 Pixel Press Technology, LLC Smart video game board system and methods
US9579573B2 (en) * 2013-06-10 2017-02-28 Pixel Press Technology, LLC Systems and methods for creating a playable video game from a three-dimensional model
JP6398248B2 (en) * 2014-01-21 2018-10-03 セイコーエプソン株式会社 Position detection system and method for controlling position detection system
GB2532075A (en) * 2014-11-10 2016-05-11 Lego As System and method for toy recognition and detection based on convolutional neural networks
US10511895B2 (en) 2015-10-09 2019-12-17 Warner Bros. Entertainment Inc. Cinematic mastering for virtual reality and augmented reality
CN108984293B (en) * 2017-06-01 2021-12-21 网易(杭州)网络有限公司 Game content display method and device, storage medium, processor and terminal

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080113581A1 (en) * 2006-11-13 2008-05-15 Peter Jazenski Convertible infant activity center
US20090291764A1 (en) * 2008-05-21 2009-11-26 George Francis Kirkman System and apparatus for a controlled toy to interact with a computer game
US20110098092A1 (en) * 2009-10-27 2011-04-28 Reiche Iii Paul Video game with representative physical object related content

Family Cites Families (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB1332857A (en) 1971-05-17 1973-10-10 Entryown Patents Ltd Foldable articles
US5534917A (en) * 1991-05-09 1996-07-09 Very Vivid, Inc. Video image based control system
US5528263A (en) * 1994-06-15 1996-06-18 Daniel M. Platzker Interactive projected video image display system
NZ291950A (en) * 1994-07-28 1998-06-26 Super Dimension Inc Computerised game board: location of toy figure sensed to actuate audio/visual display sequence
US20020036617A1 (en) * 1998-08-21 2002-03-28 Timothy R. Pryor Novel man machine interfaces and applications
US6950534B2 (en) * 1998-08-10 2005-09-27 Cybernet Systems Corporation Gesture-controlled interfaces for self-service machines and other applications
US6681031B2 (en) * 1998-08-10 2004-01-20 Cybernet Systems Corporation Gesture-controlled interfaces for self-service machines and other applications
US7252588B2 (en) * 1999-12-01 2007-08-07 Namco Bandai Games Inc. Sport video game machine and play control method of sport video game
US6663491B2 (en) * 2000-02-18 2003-12-16 Namco Ltd. Game apparatus, storage medium and computer program that adjust tempo of sound
US7445550B2 (en) * 2000-02-22 2008-11-04 Creative Kingdoms, Llc Magical wand and interactive play experience
US7227526B2 (en) * 2000-07-24 2007-06-05 Gesturetek, Inc. Video-based image control system
US6761634B1 (en) * 2001-06-07 2004-07-13 Hasbro, Inc. Arcade table
US8316298B2 (en) 2002-06-18 2012-11-20 Aaron Conti Method and apparatus for providing entertainment through an interactive device
US6890179B2 (en) * 2002-10-08 2005-05-10 Cashflow Technologies, Inc. Interactive games for teaching financial principles
US7534157B2 (en) 2003-12-31 2009-05-19 Ganz System and method for toy adoption and marketing
JP3648535B1 (en) * 2004-03-31 2005-05-18 コナミ株式会社 GAME DEVICE, COMPUTER CONTROL METHOD, AND PROGRAM
US7394459B2 (en) * 2004-04-29 2008-07-01 Microsoft Corporation Interaction between objects and a virtual environment display
JP4005060B2 (en) * 2004-06-30 2007-11-07 株式会社ソニー・コンピュータエンタテインメント Information processing system, program, and game character movement control method
JP3746060B2 (en) * 2004-07-20 2006-02-15 コナミ株式会社 GAME DEVICE, COMPUTER CONTROL METHOD, AND PROGRAM
US8144118B2 (en) * 2005-01-21 2012-03-27 Qualcomm Incorporated Motion-based tracking
ATE554837T1 (en) * 2005-06-20 2012-05-15 Rietzler Manfred ARRANGEMENT FOR THE ELECTRONIC CONDUCT OF BOARD, ROLLER OR TOLLING CARD GAMES
JP4810573B2 (en) * 2005-08-11 2011-11-09 エヌ−トリグ リミテッド Apparatus for detecting object information and method of using the same
EP2584530A2 (en) * 2006-08-03 2013-04-24 Alterface S.A. Method and device for identifying and extracting images of multiple users, and for recognizing user gestures
US8264493B2 (en) * 2008-05-12 2012-09-11 Playcast Media Systems, Ltd. Method and system for optimized streaming game server
US8154553B2 (en) * 2008-05-22 2012-04-10 Playcast Media System, Ltd. Centralized streaming game server
US10265609B2 (en) * 2008-06-03 2019-04-23 Tweedletech, Llc Intelligent game system for putting intelligence into board and tabletop games including miniatures
US8602857B2 (en) * 2008-06-03 2013-12-10 Tweedletech, Llc Intelligent board game system with visual marker based game object tracking and identification
EP2157545A1 (en) 2008-08-19 2010-02-24 Sony Computer Entertainment Europe Limited Entertainment device, system and method
US8057290B2 (en) * 2008-12-15 2011-11-15 Disney Enterprises, Inc. Dance ring video game
EP2357605B1 (en) * 2009-12-28 2013-01-16 Softkinetic Software Stabilisation method and computer system
FR2967804B1 (en) * 2010-11-19 2013-01-04 Total Immersion METHOD AND DEVICE FOR DETECTING AND TRACKING REAL-TIME MOVING NON-RIGID OBJECTS IN A VIDEO STREAM ENABLING A USER TO INTERACT WITH A COMPUTER SYSTEM
JP5832666B2 (en) 2011-12-20 2015-12-16 インテル・コーポレーション Augmented reality representation across multiple devices
US9033795B2 (en) * 2012-02-07 2015-05-19 Krew Game Studios LLC Interactive music game
US9649565B2 (en) * 2012-05-01 2017-05-16 Activision Publishing, Inc. Server based interactive video game with toys
US9492762B2 (en) 2012-05-08 2016-11-15 Funfare, Llc Sensor configuration for toy
US9039532B2 (en) * 2012-10-31 2015-05-26 Activision Publishing, Inc. Interactive video game with toys having functionality that is unlocked through game play
US8858339B2 (en) * 2012-12-11 2014-10-14 Activision Publishing, Inc. Interactive video game system comprising toys with rewritable memories

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080113581A1 (en) * 2006-11-13 2008-05-15 Peter Jazenski Convertible infant activity center
US20090291764A1 (en) * 2008-05-21 2009-11-26 George Francis Kirkman System and apparatus for a controlled toy to interact with a computer game
US20110098092A1 (en) * 2009-10-27 2011-04-28 Reiche Iii Paul Video game with representative physical object related content

Also Published As

Publication number Publication date
US9011246B1 (en) 2015-04-21

Similar Documents

Publication Publication Date Title
US9747307B2 (en) Systems and methods for immersive backgrounds
US9011246B1 (en) Systems and methods for immersive backgrounds
JP6746801B2 (en) Creation, broadcasting, and viewing of 3D content
US10445941B2 (en) Interactive mixed reality system for a real-world event
EP2992682B1 (en) Content generation for interactive video projection systems
US9833698B2 (en) Immersive storytelling environment
CN105210373B (en) Provide a user the method and system of personalized channels guide
JP2020072841A (en) Filtering and parental control method for limiting visual operation on head-mounted display
KR20200127172A (en) Player-to-audience handoff and other audience control
US20170212771A1 (en) Distribution and custom display of graphical components
US20130154958A1 (en) Content system with secondary touch controller
CN104937641A (en) Information processing device, terminal device, information processing method, and programme
CN103357177A (en) Using a portable gaming device to record or modify a game or application in real-time running on a home gaming system
CN106462324A (en) A method and system for providing interactivity within a virtual environment
KR101572346B1 (en) Service system and service method for augmented reality stage, live dance stage and live audition
US10916061B2 (en) Systems and methods to synchronize real-world motion of physical objects with presentation of virtual content
US10642360B2 (en) Methods, systems, and computer readable media involving a content coupled physical activity surface
US20190043263A1 (en) Program executed on a computer for providing vertual space, method and information processing apparatus for executing the program
CN107708819A (en) Response formula animation for virtual reality
US20190215582A1 (en) Smart furniture content interaction system and method
US20210216191A1 (en) Distally shared, augmented reality space
CN104994920B (en) Presenting has the function of the digital content item of layer-stepping
US20170246534A1 (en) System and Method for Enhanced Immersion Gaming Room
US20220254082A1 (en) Method of character animation based on extraction of triggers from an av stream
US11402917B2 (en) Gesture-based user interface for AR and VR with gaze trigger

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14862440

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14862440

Country of ref document: EP

Kind code of ref document: A1