WO2007130641A2 - Lighting control of a user environment via a display device - Google Patents
Lighting control of a user environment via a display device Download PDFInfo
- Publication number
- WO2007130641A2 WO2007130641A2 PCT/US2007/010944 US2007010944W WO2007130641A2 WO 2007130641 A2 WO2007130641 A2 WO 2007130641A2 US 2007010944 W US2007010944 W US 2007010944W WO 2007130641 A2 WO2007130641 A2 WO 2007130641A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- user environment
- frame area
- user
- computing device
- outer frame
- Prior art date
Links
- 238000000034 method Methods 0.000 claims abstract description 50
- 230000005540 biological transmission Effects 0.000 claims abstract description 26
- 230000004044 response Effects 0.000 claims abstract description 22
- 238000004891 communication Methods 0.000 claims description 28
- 230000008859 change Effects 0.000 claims description 7
- 230000003247 decreasing effect Effects 0.000 claims description 3
- 238000012545 processing Methods 0.000 description 36
- 239000003086 colorant Substances 0.000 description 17
- 238000007726 management method Methods 0.000 description 14
- 230000002452 interceptive effect Effects 0.000 description 13
- 230000000007 visual effect Effects 0.000 description 12
- 230000003993 interaction Effects 0.000 description 11
- 230000000694 effects Effects 0.000 description 10
- 125000001475 halogen functional group Chemical group 0.000 description 10
- 238000005259 measurement Methods 0.000 description 9
- 230000002411 adverse Effects 0.000 description 8
- 238000005286 illumination Methods 0.000 description 8
- 230000001276 controlling effect Effects 0.000 description 7
- 238000013481 data capture Methods 0.000 description 6
- 230000033001 locomotion Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000002829 reductive effect Effects 0.000 description 6
- 230000006835 compression Effects 0.000 description 5
- 238000007906 compression Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 238000004513 sizing Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000007935 neutral effect Effects 0.000 description 3
- 238000004321 preservation Methods 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 108091006146 Channels Proteins 0.000 description 2
- 208000012886 Vertigo Diseases 0.000 description 2
- 239000000654 additive Substances 0.000 description 2
- 230000000996 additive effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000004069 differentiation Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- 235000008694 Humulus lupulus Nutrition 0.000 description 1
- 241000023320 Luma <angiosperm> Species 0.000 description 1
- 241001085205 Prenanthella exigua Species 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000593 degrading effect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000001037 epileptic effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000000945 filler Substances 0.000 description 1
- 235000020280 flat white Nutrition 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000009738 saturating Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 231100000889 vertigo Toxicity 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
- H04N5/2226—Determination of depth image, e.g. for foreground/background separation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/4104—Peripherals receiving signals from specially adapted client devices
- H04N21/4131—Peripherals receiving signals from specially adapted client devices home appliance, e.g. lighting, air conditioning system, metering devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42202—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/4223—Cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/74—Circuitry for compensating brightness variation in the scene by influencing the scene brightness using illuminating means
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/147—Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
Definitions
- the present invention generally relates to the generation of visual data through an image capture device during an audio/visual session such as an
- the present invention relates to control of lighting conditions in a user environment whereby the bandwidth consumed by transmission of image data generated and/or captured during such audio/visual sessions may be effectively managed. Further, " the present invention relates to control of lighting conditions in a user environment that affect the quality of image data generated during audio/visual sessions.
- Video image processing translates the movement of the user that has been captured as a sequence of video images into signals that may be used, for example, to control a character or avatar in a video game. Alternatively, image processing may generate a video image to be displayed in a 'chaf session in a manner similar to a video conference.
- An image capture device typically scans a field of view in which a user may be present (e.g., a user environment such as an office, game room, living room or the like). The captured video image may be applied to a video digitizer that provides digital output to a processor that analyzes and processes the digital information received from the digitizer.
- the processor Based upon the position or movement of the participant in the field of view, the processor produces signals that are used by the graphics generating system to move objects on the display. Similarly, the system may generate an image of the user for transmission to another user in a chat session.
- the output of the devices on graphical displays can be significantly affected by a variety of factors, especially lighting conditions in the user environment.
- the computer processing time required for image processing in an ideal user environment may be extensive and complex and tends to require substantial computing and/or temporal resources.
- a user environment that is overexposed or underexposed as to particular light sources only complicates such processing activities in that the system must compensate for the adverse lighting conditions thereby resulting in slower computing operations and affecting any realtime interactions.
- the lighting conditions are so adverse that a computing system cannot compensate for the particular environment conditions and inaccurate data— if any— is generated thereby resulting in incorrect game control or the generation of poor chat session video.
- lighting conditions may contribute to the amount of bandwidth consumed by transmission of a captured video image (or images). This may be especially in the context of real-time, full-motion video as may be generated in the context of a video chat session.
- Video signals by their very nature, consume large amounts of bandwidth notwithstanding the application of a variety of compression techniques (e.g., lossy compression) as are known in the art.
- An overwhelming presence of a certain color or colors in a particular environment e.g., as might be generated by bright lights of a particular color
- Transmission of that image and its excessive color saturation may increase an overall file size and lend to excessive bandwidth consumption. Excessive bandwidth consumption may, in turn, contribute to a degraded conferencing experience (e.g., jitter).
- An embodiment of the present invention provides a lighting control system for implementation in a variety of user environments, comprising an image capture device configured to capture image data in the user environment, the image capture device further configured to identify at least one lighting condition in the user environment.
- a computing device may be configured to generate an inner frame area and an outer frame area on a display device.
- the inner frame area includes image data captured by the image capture device and the outer frame area controls at least one lighting condition in the user environment.
- a further embodiment of the present invention provides a method for controlling lighting conditions in a user environment. The method includes the capture of image data in the user environment and the identification of at least one lighting condition in that environment.
- An inner frame area and an outer frame area are displayed on a display device whereby at least one lighting condition in the user environment may be controlled through various manipulations of the outer frame area.
- a machine- readable medium comprising an executable program may be embodied in the medium, the program comprising instructions for controlling lighting conditions in the user environment.
- Those instructions include a method comprising the capture of image data in the user environment; identification of at least one lighting condition in the user environment; and the generation of an inner frame area and an outer frame area on a display device. At least one lighting condition may be controlled in the user environment through various manipulations of the outer frame area.
- a still further embodiment of the present invention provides a video conference system.
- the system comprises a first image capture device configured to capture first user environment image data in a first user environment for transmission over a communications network.
- a second image capture device may be similarly configured for a similar purpose in a second user environment.
- a conference management server may be configured to receive first and second user environment image data and to identify at least one lighting condition in the two environments.
- Client computing devices associated with the respective image capture devices receive lighting control instructions from the conference management server whereby an outer frame area and an inner frame area are displayed on a display device at each conferencing environment.
- the outer frame area displayed at each environment may be manipulated such that at least one lighting condition in the respective user environment may be subject to control or modification by the conferencing server.
- a system for managing bandwidth consumption through control of lighting conditions in a user environment is disclosed.
- An image capture device captures image data in a user environment, which is then transmitted over a communications network.
- a computing device may be coupled to the communications network to measure consumption of bandwidth related to transmission of the captured image data over the communications network.
- a display device may display an inner frame area and an outer frame. The inner frame area and the outer frame area control at least one lighting condition in the user environment in response to the consumption of bandwidth as measured by the computing device.
- a method for managing bandwidth consumption through control of lighting conditions in a user environment is disclosed.
- Image data is captured and transmitted over a communications network. Consumption of bandwidth related to transmission of the captured image data over the communications network is measured.
- An inner frame area and an outer frame area may be generated on a display device in the user environment. At least one lighting condition in the user environment may be controlled via the inner frame area and the outer frame area in response to the consumption of bandwidth as measured by the computing device.
- a video conference system includes a first image capture device and a second image capture device. Each device is configured to capture user environment image data in a respective user environment and for transmission over a communications network.
- a conference management server receives the first user environment image data and the second user environment image data and measures consumption of bandwidth related to transmission of the first user environment image data and the second user environment image data over the communications network.
- a first user client computing device and a second user client computing device are each associated with a respective image capture device.
- the first and second user client computing devices receive lighting control instructions from the conference management server in response to the consumption of bandwidth as measured by the conference management server.
- the lighting control instructions received from the conference management server include generating an outer frame area and an inner frame area on a display device at both the first user client computing device and second user client computing device, the outer frame area and inner frame area collectively controlling at least one lighting condition in the respective user environment.
- FIGURE 1 illustrates an exemplary system for generating real-time, three-dimensional, interactive environment data according to an embodiment of the present invention.
- FIGURE 2 illustrates a block diagram of one embodiment of a client computing device for generating real-time, three-dimensional, interactive environment data.
- FIGURES 3A and 3B illustrate an exemplary video display device comprising an inner frame area and an outer frame area for implementing lighting control of a user environment in accordance with one embodiment of the present invention.
- FIGURES 4A and 4B illustrate an exemplary video display device reflecting different degrees of illumination intensity in the outer frame in accordance with one embodiment of the present invention.
- FIGURE 5A illustrates an exemplary method for implementing lighting control of a user environment through varying illumination intensity in a video display device in accordance with one embodiment of the present invention.
- FIGURE 5B illustrates an exemplary method for managing bandwidth with respect to the generation and transmission of image capture data in accordance with one embodiment of the present invention.
- FIGURE 6 illustrates an exemplary network of real-time, three- dimensional, interactive environment data generating systems as may be utilized in a 'chat' session or interactive video game in accordance with one embodiment of the present invention.
- FIGURE 1 illustrates an exemplary system 100 for generating realtime, three-dimensional, interactive environment data according to an embodiment of the present invention.
- the system 100 comprises an image capture device 110, a video display device 120, and a client computing device 130 having processor functionality such as a home entertainment system for video games.
- the client computing device 130 may be further coupled to a controller device 140 as is shown in FIGURE 1.
- a user and a related environment such as a living room, are located within the field-of-view of the image capture device 110.
- the client computing device 130 may be implemented as a
- client computing device 130 and its processor functionality may be implemented in other types of computing devices such as personal computers, workstations, laptop computers, wireless computing devices, portable media devices, or any other type of computing device that may be capable of receiving and processing graphical image data.
- Video display device 120 may be any device configured for the display of video data such as a television or computer monitor.
- video display device 120 receives graphical image data from client computing device 130 via an AV MULTI OUT connector that may be coupled to an integrated audio/video cable and the requisite video and audio connectors.
- Various other connection mechanisms may also be used, for example, S VIDEO, an RFU adaptor, component video input (Y CB/PB CR/PR), and HDTV input (Y PB PR) as well as the High Definition Multimedia Interface (HDMI) and Digital Visual Interface (DVI).
- the client computing device 130 and video display device 120 may be integrated as is the case in a PlayStation® Portable from Sony Computer Entertainment Inc.
- Image capture device 110 may be, in one embodiment of the present invention, a color digital camera device with functionality similar to that of a webcam.
- Image capture device may be coupled to client computing device 130 via a USB cable.
- image capture device 110 may utilize a wireless transmission protocol to exchange data with the client computing device 130 (e.g., 802.1Ix).
- Image capture device may also include commercially available dedicated video conferencing products such as those manufactured by Polycom, Inc. of Pleasanton, California.
- Image capture device 110 extracts specific information from captured multi-dimensional image data for subsequent presentation to a user of the device (e.g., a color image on visual display device 120 or for transmission to another user over a network as in FIGURE 6 below).
- captured image data may be integrated into a video game or other user-centric application (e.g., a workout or training video).
- Captured image data may also be utilized in a video conference or 'audio/visual chat session' wherein the captured data may be transmitted to another user's computing device for display.
- Image capture device 110 may be capable of capturing and mapping in three-dimensions in addition to normal two-dimensional video imagery. Similar to normal cameras, image capture device 110 captures two-dimensional data for a plurality of pixels that comprise the video image. These values are color values for each pixel— generally red, green, and blue (RGB) values. In this manner, objects captured by the image capture device 110 appear as two-dimensional objects on a monitor.
- RGB red, green, and blue
- image capture device 110 may also capture depth values in a particular field-of-view (e.g., a particular user environment). That is, image capture device 110 may capture the x and y components of an environment using RGB values for each pixel in the environment in addition to a z-component, which represents a depth value for the environment. (i.e., the z-axis).
- a z-value may be captured for each pixel of the scene, each z-value representing a distance from the camera to a particular object in the scene corresponding to the related pixel.
- a maximum detection range may be defined beyond which depth values will not be detected.
- An input image processor (not shown) at client computing device 130 translates the captured video images and depth data into signals that are delivered to an output image processor (not shown) at the client computing device 130.
- the output image processor may be programmed to effect movement and status of virtual objects on the video display device 120 in response to signals received from the input image processor.
- input and output image processors may be an integrated part of the hardware and software configuration of the image capture device 110.
- processing functionality may be partially distributed over a network like that shown in FIGURE 6, such processing taking place at, for example, a conferencing management server.
- Some embodiments of the image capture device 110 may also be configured for the capture and subsequent processing of visual data in high-definition.
- Various image processing techniques allow a user to interact with the image capture device 110 device using motion, color detection and, in some embodiments, sound through a built-in microphone or other audio input device (not shown) coupled to the image capture device 110 or client computer device 120.
- Certain interaction techniques are disclosed in U.S. patent publication number US 2004-0207597 Al for a "Method and Apparatus for Light Input Device” and U.S. patent number 7,102,615 for a "Man-Machine Interface Using a Deformable Device.”
- a prop-input device is disclosed in U.S. patent publication number US 2004-0239670 Al for a "System and Method for Providing a Real-T ⁇ me Three-Dimensional Interactive Environment" as well as U.S.
- the image capture device liO may be compact in design allowing for placement on top of a video console or television.
- Image capture device 110 may comprise a pivot thereby allowing for positioning of the device and its related field-of-view.
- the image capture device 110 may further comprise a rotational ring around the camera lens for manual focus control.
- Some embodiments of the image capture device 110 may provide for automated positioning and focusing through the use of a directional-sensitive microphone that tracks a source of audio (e.g., an individual who is speaking) and the focus of the device 110 may be subsequently adjusted according to the position of the audio source relative to the image capture device 110.
- a source of audio e.g., an individual who is speaking
- Client computing device 200 may be used to aid in the generation of real-time, three-dimensional, interactive environment data.
- the client 200 may be communicatively coupled to image capture device 110 in order to generate the aforementioned environment data.
- the client 200 may comprise, but is not limited to, a main memory
- the client 200 may also comprise an IOP memory 214, a controller interface 216, a memory card 218, a Universal Serial Bus (USB) interface 220, and an IEEE 1394 interface 222.
- the client 200 may further include an operating system read-only memory (OS ROM) 224, a sound processing unit (SPU) 226, an optical, disc control unit 228, and a hard disc drive (HDD) 230, all of which may be connected via a bus 238 to IOP 212.
- OS ROM operating system read-only memory
- SPU sound processing unit
- HDD hard disc drive
- client 200 may also include a network adaptor 240, which may offer an Ethernet connection 242 and/or telephony connection 244.
- the client 200 in one embodiment, may be an electronic gaming console although client 200 (or portions thereof) may also be implemented as a general-purpose computer, a set-top box, a hand-held gaming device, or in a mobile device such as a cellular phone.
- client 200 or portions thereof may also be implemented as a general-purpose computer, a set-top box, a hand-held gaming device, or in a mobile device such as a cellular phone.
- system architectures may be utilized within the scope of the present invention such as the computer architecture and high speed processing model disclosed in U.S. patent publication number 2002-0138637 for a "Computer Architecture and Software Cells for Broadband Networks," the disclosure of which is incorporated herein by reference.
- the CPU 206 communicates with the main memory 202 via a dedicated bus 234.
- the VUl 208 and the GPU 210 may also communicate with one another via a dedicated bus 232.
- the CPU 206 executes programs stored in the OS ROM 224 and the main memory 202.
- the main memory 202 may contain pre-stored programs and may also contain programs transferred via the IOP 212 from a CD-ROM, DVD-ROM, or other optical disc (not shown) using the optical disc control unit 228.
- the IOP 212 controls data exchanges between the CPU 206, the VUO 204, the VUl 208, the GPU 210 and other devices of the system 200, such as the controller interface 216, or from other such systems via the network adaptor 240.
- the GPU 210 executes drawing instructions from the CPU 206 and the
- VUO 204 to produce images for display on a display device (not shown).
- the VUl 208 transforms objects from three-dimensional coordinates to two-dimensional coordinates, and sends the two-dimensional coordinates to the GPU 210.
- the SPU 226 executes instructions and processes data to produce sound signals that are output on an audio device (not shown).
- a user of the client 200 provides instructions to the CPU 206 via a controller coupled to the controller interface 216.
- Controller may be any control device, for example, a joystick, a set of directional buttons, and/or other control buttons.
- An exemplary controller is illustrated in FIGURE 1 (140).
- a user may instruct the CPU 206 to store certain information on the memory card 218, which may be removable (e.g., a flash memory or other non-volatile memory card), or may instruct a character in a game to perform some specified action.
- Other devices may be connected to the system 200 via the USB interface 220 and the IEEE 1394 interface 222.
- the image capture device 110 may be coupled to the client 200 utilizing a USB connector or, in another example, a wireless Ethernet network through Ethernet connection 242.
- the client 200 comprise a network adaptor 240 that provides the hardware functionality necessary for the client 200 to connect to a network.
- the network adaptor 240 may comprise, for example, a system connector that operates to connect the adaptor 240 to the client computing device 200 through an expansion bus connector 246.
- the network adaptor 240 may also comprise a power connector and data connector to allow for the provisioning of power from the client 200 to the adaptor 240 and the exchange of data between the client 200 and the adaptor 240.
- Network adaptor 240 may be fully integrated with the client 200 or may be a detachable hardware device that may be implemented in older legacy client devices 200.
- the network adaptor In some embodiments of the present invention, the network adaptor
- 240 may also require the installation of certain software on the client 200 to allow for identification and connection to a particular IP address and/or dial-up to a particular Internet Service Provider.
- Software may also provide other functionalities, such as the creation and maintenance of user profiles, in addition to functional interaction between the client 200 and the network adaptor 240.
- Such software or data related to such functionality may be embodied on CD-ROMs for games or application requiring a network connection; stored on memory card 218; or part of a firmware upgrade.
- the network adaptor 240 may also comprise an Ethernet connection
- a network cable (e.g., a 100 Base-TX or 10- Base T) may be coupled to the network adaptor 240 for connection to a network.
- the network cable may, for example, be communicatively coupled to a DSL or cable modem.
- the network cable may also be communicatively coupled to, for example, a router via a LAN port; the router may then be coupled to a DSL or cable modem through a WAN port.
- the Ethernet connection 242 may allow for a network cable to be connected to a wireless Ethernet bridge.
- the wireless Ethernet bridge may be communicatively coupled to a wireless router utilizing, for example, an 802. Hx protocol.
- the wireless router may be further communicatively coupled to a cable or DSL modem.
- the network adaptor 240 may also comprise a telephony connection
- FIGURE 3A illustrates an exemplary video display device 300 comprising an inner frame area 310 and an outer frame area 320 for implementing lighting control of a user environment in accordance with one embodiment of the present invention. Lighting control of the user environment may also be used in the context of bandwidth management as further discussed herein.
- Video display device 300 corresponds to the video display device 120 originally referenced in FIGURE 1 and coupled to client computing device 130 (client 200 of FIGURE 2) for the display of images or other drawing instructions by GPU 210.
- Video display device 300 may also comprise audio output functionality ⁇ e.g., a built-in speaker system) to emit audio signals produced by SPU 226.
- Inner frame area 310 and outer frame area 320 are artificial boundaries that may be created by client computing device 130 as the result of drawing instructions from the CPU 206 being executed by GPU 210.
- the dimensions of these areas may be determined by or in light of a particular software title being executed (e.g., a video game title utilizing an image capture device 110) or by a server component as might be utilized in an audio/visual 'chaf session as is discussed in further detail in FIGURE 6.
- the outer frame area 320 may be the extra 'dead space' on the sides of a widescreen display.
- the size of outer frame area 320 is larger than that of the corresponding frame area 320 in FIGURE 3A. Accordingly, the inner frame area 310 of FIGURE 3B is smaller than that of the corresponding frame area 310 in FIGURE 3A.
- the inner frame 310 and outer frame 320 may also be subject to particular user configurations or settings that may be saved on memory card 218 or some other memory device coupled to the client computing device 130. Inner frame 310 and outer frame 320 may be adjusted in response to certain determinations made by client computing device 130 or in response to certain instructions received from another computing device over a network. [0046] In one embodiment of the present invention, video game data will be displayed in the inner frame area 310.
- the image capture device 110 may capture an image of the user for insertion into a game environment in the form of an avatar.
- the image capture device 110 may generate data used to determine whether the user came into contact with a particular portion of the game environment by 'grabbing' at an object or providing input data through a prop device.
- This real-world activity or interaction may be reflected in the game environment as a result of the user, for example, swinging at a baseball with a baseball bat prop device in an interactive baseball game.
- An image capture device 110 tracks and captures the swinging of the baseball bat prop device and, through the necessary images processors, translates that real-world activity into on-screen display data.
- the inner frame area 310 may also be used to display 'chaf session data such as the image of a remote participant communicating with the user.
- the inner frame area 310 may also display multi-media data generated by the remote participant and transmitted to the present user in the form of a textual 'instant message' or a short video clip.
- Various types of interactive data may also be displayed in the inner frame area 310 such as an interactive game (e.g., chess or checkers) or a collaborative document (e.g., a report that is being concurrently edited by one or more parties).
- image capture device 110 may require certain light levels or light conditions in order for the image capture device 110 to function properly or optimally. If the environment in which the image capture device 110 is operating in lacks sufficient light, certain data (e.g., user interactions or prop input data) may not be detected because the image capture device 110 cannot properly discern various x-y-z position values, color differentiations, or otherwise detect motion of a user or object. Image capture device 110, in conjunction with software and/or hardware operating internal to the image capture device 110, at the client computing device 130, and/or by an intermediate server as is discussed in FIGURE 6, will determine whether there is proper light present in the environment to properly capture data (e.g., with respect to a target histogram).
- data e.g., user interactions or prop input data
- Image capture device 110 in conjunction with software and/or hardware operating internal to the image capture device 110, at the client computing device 130, and/or by an intermediate server as is discussed in FIGURE 6, will determine whether there is proper light present in the environment to properly capture data (e.g., with respect
- IR infrared
- Various other spectral emissions may be subject to such user environment light control.
- determinations may be made with respect to the presence of too much light or over-saturation with respect to a particular color. These excesses may result in spikes in bandwidth consumption when transmitting image or video data captured and/or generated by image capture device 110. Spikes in bandwidth consumption may result in a less than optimal conferencing experience, especially in constricted bandwidth environments.
- Image capture device 110 in conjunction with the aforementioned software and/or hardware operating internal to the image capture device 110, at the client computing device 130, and/or by an intermediate server as discussed in FIGURE 6 may determine, for example, that too much red light is present in the environment. This light may come from an external source such as natural sunlight through a window proximate the user environment, from an artificial lighting source in the user environment (e.g., an overhead lamp), or from the display device 300 itself.
- an external source such as natural sunlight through a window proximate the user environment
- an artificial lighting source in the user environment e.g., an overhead lamp
- certain colors may be 'muted/ 'countered' or otherwise 'cancelled' through additive (e.g., mixing of red, green, and blue) and subtractive (e.g., subtracting a hue out of a color scheme by adding more of another) color techniques as are known in color theory.
- additive e.g., mixing of red, green, and blue
- subtractive e.g., subtracting a hue out of a color scheme by adding more of another
- the outer frame area 320 may be used to provide additional light to the user environment such that the video display device 300 operates as an artificial light source (e.g., certain RGB values of f individual pixels in the display 300 are adjusted as to average a necessary user environment light level). That is, the outer frame area 320 will become visibly brightened with a particular color and/or intensity of light such that the video display device 300 lights the environment proximate the image capture device 110 whereby image data may be properly captured for subsequent processing.
- an artificial light source e.g., certain RGB values of f individual pixels in the display 300 are adjusted as to average a necessary user environment light level. That is, the outer frame area 320 will become visibly brightened with a particular color and/or intensity of light such that the video display device 300 lights the environment proximate the image capture device 110 whereby image data may be properly captured for subsequent processing.
- a user may be utilizing the image capture device 110 to play the aforementioned simulated baseball game with a prop device.
- the user may find an avatar of themselves inserted into the 'home plate' video game environment and displayed in the inner frame area 310 of FIGURE 3A.
- the video game software operating at client computing device 130 may (via CPU 206) cause the execution of rendering instructions by the GPU 210 that causes the rendering on the display device 300 of a white 'filler' in the outer frame area 320 (i.e., the execution of drawing instructions that causes the individual pixels within the outer frame area 320 to display a particular luminance and/or shading).
- the inner frame area 310 would continue to display the baseball video game and related information while simultaneously becoming surrounded by a white 'halo' in the outer frame area 320.
- This white 'halo' displayed by the video display device 300 will 'spill' into the real- world environment of the user —that is, the light emitted by the display and measured in candelas per meter squared (cd/m 2 )— whereby the video display device 300 actually becomes an alternative light source.
- This additional light generated by the video display device 300 will result in the generation of sufficient additional light to allow for the image capture device 110 to properly capture image data for processing.
- the intensity of the 'halo' emitted by the outer frame area 320 can be controlled in part by the particular rendering instructions from GPU 210; that is, how white (what shade of white) and how bright (the intensity thereof).
- the color and/or intensity of the 'halo' of the outer frame area 320 may be adjusted and manipulated to compensate for the same.
- excess saturation also referred to as chroma
- red may be countered by a hue of the opposite color (blue/green).
- the particular saturation / hue combination may be wholly opposite, slightly opposite, or substantially opposite the excess saturation depending on particular network conditions.
- the particular hardware configuration of the display device 300 may be taken into account such that the GPU 210 may optimize the resources at its disposal with regard to manipulating the outer frame area 320 and the light emitted by the same.
- These configurations may be automatically detected by the client computing device 200 or provided to the device 200 through user input as to particular configurations, the brand and model of the display device (which in turn corresponds to particular operating specifications), and so forth.
- device 200 may consult a look-up table or other data store (either locally or over a network) to identify particular operating capabilities (e.g., bit depth) for a particular display model if those capabilities are not immediately identified by the user or through certain plug-and-play (PnP) functionality.
- PnP plug-and-play
- a liquid crystal display (LCD) in a flat screen television comprises a certain resolution (i.e., the number of dots of color — pixels — on the display) referenced by a horizontal axis (rows) and vertical axis (columns).
- a wide aspect 22-inch LCD monitor may conform to the WUXGA (Wide Ultra Extended Graphics Array) standard and comprise a resolution of 1920x1200 whereas a smaller device such as a 15-inch LCD monitor may only conform to the XGA (Extended Graphics Array) standard and comprise a resolution of 1024x768. While an LCD display device is referenced, the present invention is equally as applicable to a CRT- based display.
- a display that is configured to operate in SuperVGA mode can display 16,777,216 colors (also known as true color) the result of a 24-bit-long description of a pixel (the bit depth). For example, in 24-bit bit depth, eight bits are dedicated to each of the three additive primary colors (RGB). The bit depth therefore determines the number of colors that can be displayed at one time.
- RGB additive primary colors
- the brightness of the light emitted into the user environment can also be affected by the size of the outer frame area 320 relative the inner frame area 310.
- a certain degree of luminance may be created.
- Identical white image data i.e., the same intensity and shade
- FIGURE 3B will create a greater degree of luminance in that the outer frame area 320 of FIGURE 3B is larger than that of FIGURE 3A.
- the amount of light that is generated by the video display device 300 though a 'halo' may be dynamically controlled during the use of image capture device 110.
- the image capture device 110 and related client device are identical to the image capture device 110 and related client device.
- the outer frame area 320 may, at that point, be illuminated as a white halo to provide the additional light necessary to allow for game play to continue.
- the intensity of the light may be controlled by the client computing device 130 that may be running game software that has been calibrated with the image capture device 110 to determine when the proper amount of light is present to allow for the game to function as intended. In that regard, requisite light settings may be predetermined or calibrated based on the particular game and/or user environment.
- the intensity of the artificial light source emitted from the display device 300 may be gradually increased.
- a dull gray light source in the outer frame area 320 may provide sufficient light at 4.00 PM but a flat white emission may be required from the outer frame area 320 at 6.00 PM.
- the intensity of the light may also be increased (e.g., the brightness of the light).
- the limitations of the video display device 300 may be such that even the most intense white available does not provide sufficient 'artificial lighf to allow game play to continue.
- the GPU 210 may cause for the size of the inner frame area 310 to decrease and the outer frame area 320 to increase thereby providing additional artificial light (i.e., additional individual pixels in the screen display utilized for illumination purposes versus the display of game data).
- additional artificial light i.e., additional individual pixels in the screen display utilized for illumination purposes versus the display of game data.
- particular portions of the user environment may be subject to insufficient lighting conditions.
- the right side of a room may have an open window whereby sufficient light may be provided to allow for image data capture.
- the left side of the room may have window shades drawn such that no natural light may be entering that portion of the user environment.
- the orientation of the inner frame area 310 and outer frame 320 may be adjusted such that the inner frame area 310 may be moved to the far right hand side of the display 300 whereby the inner frame area 310 may be off-center but providing a larger, focused outer frame area 320 on the left-side of the display such that additional artificial light may be generated by the display 300.
- the positioning and centering of the frame areas 310 and 320 may be dynamically adjusted as user of the image capture device 110 continues.
- particular gradations of color from one-side of the display 300 to another may be implemented wherein the orientation of the inner frame area 310 and the outer frame area 320 may remain constant.
- the portion of the outer frame area 320 relative the left side of the room may emit a white light while the right side of the outer frame area 320 displays a neutral background color in that sufficient natural light is present on the right side of the room.
- a gradual gradation of color (from white to black) between the left side of the outer frame 320 to the right side may bridge the two 'sides' of the outer frame area 320.
- Particular portions of the outer frame area 320 may be subdivided into various sub-sections. Each of those sub-sections may be subject to various lighting controls.
- the outer frame area 320 may be sub-divided into four quadrants: an upper right, upper left, lower right, and lower left quadrant.
- Each of those individual quadrants may be subject to various lighting controls (e.g., the upper right and lower left corner of the frame may be illuminated while the upper left and lower right are not).
- These various controls include color, luminance, and any other condition that may be controlled by the various systems and methods disclosed herein.
- the various sub-divisions may also be subjected to various shapes for both functional and aesthetic reasons.
- various triangle or other polygonal configurations may be implemented.
- Various circular or elliptical patterns may also be used.
- These and any other shape or design capable of being displayed e.g., lines, squiggles, waves, dots, splotches, etc.
- Various shapes and patterns may also be subject to strobing (i.e., regular, controllable series of high power flashes rather than continuous light) for a variety of effects.
- strobing may be used to create a strobe light feature in the user environment or, similarly, to counter an effect caused by natural or other artificial light conditions in the user environment.
- Certain video games or television programs that may cause epileptic fits, feelings of vertigo, and the like and that might be displayed in the inner frame area 310 may be countered through a counter- strobe effect in the outer frame area 320.
- the presently described lighting controls systems and methods may also be implemented with traditional television programming in addition to 'chat' sessions and video games.
- certain calibrations may take place through the image capture device 110.
- the image capture device 110 may, at start-up, sample the user environment to determine various lighting conditions and the effect those conditions will have on image capture over a particular operating range (e.g., lowest possible range of capture to an optimal range of capture).
- the image capture device in conjunction with the various lighting controls discussed herein as implemented by various hardware and/or software operations, may then adjust certain user environment lighting conditions through inner frame area 310 and outer frame area 320 lighting manipulations (e.g., size, brightness, orientation, color). During the course of these adjustments, the image capture device 110 may 're-sample' the environment to determine what environmental control condition will allow for a particular operation of a game or chat session in light of the particular environment.
- the computing device 130 in conjunction with particular 'chaf or video game software may make that determination or such a determination may come as a result of a management server determination as is described in the context of FIGURE 6.
- the determination of a particular user environment condition may come from a user in another environment if that particular user is unable to properly view the first user in a chat session. That decision by a second user may be made by selecting an available environment setting in light of several environmental control conditions.
- the first user environment may be subject to three different intensities of a particular color of light; three different colors of light; and three different frame area sizes.
- the image capture device 110 may capture an image of each of those nine possible environments.
- the effect of that particular environment control may then be displayed to the second user in the form of a still frame or even a short (e.g., 5 second) video clip.
- the second user may then select the particular environment control based on the image or clip that appears best.
- a user operating in a solo environment e.g., playing a video game
- adjustments may occur automatically during game play or at breaks such as between levels. Adjustments may further occur in response to a user action wherein a query may be made such as during a pause.
- a particular frame of data e.g., an I-frame
- particular frames of data may be recognized during the decoding process as likely to cause certain changes in the user environment.
- incoming image data of a conference remote participant may consist of a bright white background that will spill into the user environment thereby causing an unintended change in user environment lighting conditions.
- environment changing conditions may be identified in advance such that when the actual image data may be rendered that display 300 will have already adjusted an inner frame area 310 and/or outer frame area 320 as may be appropriate such that no adverse affect or change to the user environment occurs.
- the outer frame area 320 may always be present but simply unutilized (i.e., a neutral background color such as black). The frame area may then 'filled' as needed in response to GPU 210 executing a drawing instruction from the CPU 206. Ih other embodiments, the actual graphic images to be displayed may occupy the majority of the game screen and, as such, only the inner frame area 310 may be displayed.
- the software executed by the CPU 206 and otherwise controlling the display of the frame areas 310 and 320 and interacting with the image capture device 110 may cause the GPU 210 to reconfigure the game environment (e.g., re-size the game environment to fit in a small inner frame area 310) such that the outer frame area 320 may now be displayed and emit the appropriate 'halo' of artificial light.
- the re-sizing of the various frame areas may occur during the course of game play (e.g., the re-sizing occurs as game play progresses).
- the re-sizing may occur only during pauses in game action such as to avoid the size of the environment and various characters changing while game play is occurring (i.e., to avoid a vertigo-type effect). Additionally, re-sizing may actually cause an automatic pause in game play with a query issued to the user as to whether re-sizing is permitted. In these instances, the amount of natural light may be sufficient to allow game play to continue but not without some processing errors. If game play deteriorates to the point that additional light is necessary, the user may be asked whether they wish to activate the illumination control feature of the present invention.
- the user may manually provide another source of artificial light through, for example, an overhead lamp or other light source in the actual user environment.
- Various user input may be provided through, for example, controller 140 in addition to other inputs such as voice-recognition.
- different colors or shades of colors may also be utilized to maximize the color balance of a particular user environment. For example, a user may have different light sources offering different colors of light. These varying colors may cause difficulty with the image capture device 110 processing environment information, especially with regard to color differentiation.
- the image capture device 110 may identify irregularities or processing difficulties in the environment in a manner similar to the identification of a lack of sufficient light.
- the outer frame area 320 may used in a similar manner as to produce different colors and intensity of light to counterbalance certain overexposures in the particular environment.
- additional frame areas may be used (e.g., a third, fourth, or fifth frame area) such that combinations of certain colors (e.g., yellow and blue individual halos to create an overall green appearance) may be implemented.
- FIGURES 4A and 4B illustrate an exemplary video display device 400 reflecting different degrees of illumination intensity in the outer frame area 420 in accordance with one embodiment of the present invention.
- FIGURES 4A and 4B which reflect a 'video chaf session, illustrates the lighting adjustments of a user environment described in the context of FIGURES 3A and 3B.
- the inner frame area 410 of FIGURE 4A reflects image data related to another participant (i.e., a remote chat participant); the outer frame area 420 is presently in a neutral background state.
- the outer frame area 420 has been 'filled' such that it now reflects white light 'halo' as referenced in the context of FIGURES 3A and 3B.
- the inner frame area 410 has also been reduced in that the lighting conditions in the environment of the user receiving the present image have made it necessary to increase the amount of light emitted into that environment. As such, the outer frame area 420 was expanded at the expense of the inner frame area 410.
- the inner frame area 420 In some embodiments of the present invention, the inner frame area
- the graphic data in the inner frame area 410 may need to be reduced in that the graphic data in the inner frame area 410 is inadvertently creating excess light and thereby over saturating the user environment.
- the image displayed may pan upward to the sky to follow the path of the baseball as it travels toward the outskirts of the ballpark.
- the screen display will become predominantly white and gray (i.e., colors of the sky and clouds). These bright colors may 'spill' into the user environment, which may cause image data capture difficulties.
- the inner frame area 410 may decrease in size to reduce the amount of secondary light 'spillage' into the user environment.
- the outer frame area 420 may be darkened to help counter suddenly excess bright light being emitted from the inner frame area 410.
- FIGURE 5A illustrates an exemplary method 500 for implementing lighting control of a user environment through varying illumination intensity in a video display device in accordance with one embodiment of the present invention.
- an image capture device may attempt to capture image data from a user environment.
- a determination may be made whether adverse lighting conditions are inhibiting the processing of the image capture data or the actual capture of the data. If there are no adverse conditions present and the image data is captured and processed without incident, image data capture proceeds as necessary followed by subsequent processing of the data.
- step 520 If it is determined in step 520 that adverse lighting conditions do exist, an attempt may be made to adjust display intensity in step 530. This may occur through various adjustments whereby the outer frame area may be 'filled' with a particular color and intensity of light as discussed in the context of FIGURES 3A and 3B and 4A and 4B above. If the adjustment as to display intensity is not sufficient, then (in step 540), frame image boundaries may be enlarged (or reduced as is appropriate) in order to increase or decrease the total amount of artificial light projected into the user environment.
- step 540 and the adjustment of frame boundaries is recited as following step 530 and the adjustment of light intensity, this is not to suggest the necessity of a step-by-step process. Both adjustments may occur concurrently or frame adjustment may occur prior to intensity adjustment in that FIGURE 5A is an exemplary embodiment only. Regardless of the order, following the adjustment of image intensity and/or frame boundaries, further attempts at image capture and/or image data processing occur as the method repeats or continues (as is appropriate) in step 510.
- FIGURE 5B illustrates an exemplary method 550 for managing bandwidth with respect to the generation and transmission of image capture data in accordance with one embodiment of the present invention. Specifically, FIGURE 5B illustrates a method 550 for managing bandwidth through the implementation of lighting control of a user environment through, for example, varying illumination intensity in a video display device.
- the control of lighting conditions in a user environment may be associated with bandwidth and processing availability.
- images are represented in YUV color space (YCbCr) wherein 24-bits per pixel are present: 8-bits for luminance (Y) and 8- bits for each of the two chrominance (U), (V) elements.
- the chrominance information in the YUV color space data may be sub-sampled in both the horizontal and vertical direction. All of the luminance information may be retained, however, as the human eye is more sensitive to luminance information rather than chrominance information.
- Frames of video are subsequently divided into 16x16 macro-blocks consisting of four 8x8 luminance blocks and two 8x8 chrominance blocks (1 U and 1 V).
- Each frame of video data may then be encoded as one of three types of frames: intia-frame (I-frames), forward predicted frame (P-frames), and bidirectional predicted frames (B-frames).
- I-frames intia-frame
- P-frames forward predicted frame
- B-frames bidirectional predicted frames
- I-frame the frame may be encoded as a single image with no reference to past or future frames.
- DCT Discrete Cosine Transform
- the other frame types P-frames (relative a past reference frame) and B-frames (relative a past reference frame, a future reference frame or both)— refer to the I-frame.
- P-frames relative a past reference frame
- B-frames relative a past reference frame, a future reference frame or both
- the image data captured from the user environment could be accepted at the lowest possible operating condition. That is, lighting controls would illuminate a user environment no more so than necessary in order to reduce the amount of luminosity data being captured, compressed and subsequently transmitted over a network.
- This decreased luminosity data represents not only savings in bandwidth at a transmitting computing device, a receiving computing device, and over the network backbone but can also recognize a savings with regard to processing cycles at a computing device perform digitization and compression following image capture.
- certain colors captured from a particular user environment may be captured and/or cancelled out by adjusting display emissions and/or frame boundaries as discussed above.
- Y pertains to luminance (brightness)
- U and V are chrominance (color) components
- each channel is expressed with 8 bits (256 possible values) and uses the same amount of space.
- the human eye is most sensitive to luminance— the black and white information in picture detail. Less sensitive areas of the human eye "fill in" color. That is, color information may be reduced without the human eye noticing the same.
- the chrominance components (U) and (V) may be reduced without the human eye adversely noticing the same.
- Bandwidth consumption may be reduced by adjusting chrominance information.
- the environment may be effectively captured at (or manipulated to conform to) the equivalent of 4:2:2 format whereby the chroma components are sampled at half the rate of luma thereby reducing bandwidth of the signal by as much as one-third with little no perceptible visual difference.
- red and blue channels of the video signal may be half the bandwidth of the luminance information.
- green may be cancelled out as green can be calculated from red and blue chrominance information in the context of luminance information.
- a utility related to preservation of bandwidth and processing power vis-a-vis the image capture device and/or related hardware or software may be manually implemented by a user. For example, a user may recognize that their particular computing device lacks accelerated processing power or may be coupled to a low bandwidth network or, similarly, that a recipient of data captured by the image capture device suffers from similar limitations. In these instances, the user may activate such a utility to avoid bottlenecks of data.
- this preservation utility may be implemented on an as- needed basis, the need being recognized by a network congestion monitoring utility like ping, which utilizes the Internet Control Message Protocol (ICMP), and trace route, which utilizes the Uniform Datagram Protocol (UDP), to measure network response time.
- ICMP Internet Control Message Protocol
- UDP Uniform Datagram Protocol
- These and other network utilities may be implemented in various network devices ⁇ e.g., switches or routers) or through large-scale platforms like the Network Application Performance Analysis Solution (NAPAS) for Cisco Systems.
- NAPAS Network Application Performance Analysis Solution
- Indicia of network congestion generated by these applications and solutions may be provided to a computing device operating certain software related to image capture (e.g., a client computing device or a conference management server) such that bandwidth and/or processor preservation may be activated through the manipulation of lighting conditions that are not optimal but otherwise sufficient to allow for continued image data capture.
- a computing device operating certain software related to image capture e.g., a client computing device or a conference management server
- bandwidth and/or processor preservation may be activated through the manipulation of lighting conditions that are not optimal but otherwise sufficient to allow for continued image data capture.
- the recipient of that data may adjust the incoming video data such that it does not appear as lesser quality image data.
- the incoming video data may be pre-processed such that luminance that was degraded by the sender of the data may be artificially amplified by the recipient through editing or video preprocessing editing software that automatically adjusts the data as it is decoded and rendered.
- step 560 the bandwidth being consumed by image capture data transmitted over a network (as may occur in the context of a video chat session like that described in FIGURE 6 below) is made.
- a determination of bandwidth may be made utilizing, for example, the aforementioned NAPAS or simpler methodologies such as ping and trace route. The determination may occur at the transmitting computing device, at the receiving computing device, or at some intermediate computing device or at some other point in a communications network. The information may then be provided to the transmitting device such that it may adjust its image capture operations as is appropriate and is discussed in the context of steps 580 and 590. Alternatively, another device may make certain determination as to bandwidth consumption and provide instructions to the transmitting computing device such that it may responsively adjust its image capture operations as may occur through screen and/or frame adjustment.
- the particular bandwidth may be a static predetermined amount (e.g., a target threshold). That static threshold may be manually set by a user or network administrator. The static threshold may also be identified at the outset of a chat session and remain constant throughout the session. Alternatively, the amount may be determined in the context of a particular video chat session and may change during the course of the chat session. That is, as network conditions change from the beginning to the end of the session and at various points in between, the target threshold may be adjusted subject to those changing network conditions.
- two users chatting over a dedicated corporate network for such communications may be unlikely to encounter other users or applications attempting to utilize the same finite amount of bandwidth in the network.
- the availability of large amounts of network overhead may obviate the need to otherwise adjust network traffic consumption.
- two users are chatting over a public network shared by other users at a peak traffic time, even image capture data that is 'normal' under the circumstances may exceed a particular bandwidth allotment or may otherwise encounter transmission difficulties due to other users and applications sharing the network.
- bandwidth consumption may initially be high with respect to the group and available resources. As participants drop out of the conference (e.g., their presence is no longer required), the total bandwidth consumption may decease and what constitutes an excessive use of bandwidth for any one user may change as additional resources become available to the session.
- Measurement of bandwidth consumption may occur at predetermined times during the chat session (e.g., every five minutes). Measurement may also occur upon the occurrence of a particular event, for example, transmission of the I-frame or every n lh I-frame. When and how often measurement occurs may also be set by the user or a network administrator.
- adjustment of display intensity and frame boundaries may occur in steps 580 and 590. These particular adjustments may be similar with respect to the display and frame adjustments that occurred in steps 530 and 540 in the context of FIGURE 5A. Like FIGURE 5A, adjustment of display intensity and frame boundaries are not necessarily a step-by-step process. Both adjustments may occur concurrently or frame adjustment may occur prior to intensity adjustment. Further, only one particular adjustment (i.e., frame but not display) may occur.
- Lighting adjustment sources other than or in addition to the screen and a frame area within the screen may be utilized in the context of managing bandwidth consumed by image capture data.
- an external light source may be coupled to a device controlling lighting emissions (e.g., client computing device 130 of FIGURE 1). This coupling may occur in the context of a USB connection or other data connection.
- the external lighting device may be proximate or otherwise remote from the actual image capture device and related computing device so long as the light source may be configured to result in some influence of the lighting conditions of a particular user environment.
- Some lighting control hardware may be integrated with particular display device such as a backlight or other ambient light projector.
- FIGURE 6 illustrates an exemplary network 600 of real-time, three- dimensional, interactive environment data generating systems 610/620 as may be utilized in a 'video chat' session or interactive video game and in accordance with one embodiment of the present invention.
- Systems 610/620 exchange data over communications network 630.
- Optional conferencing management server 640 may facilitate exchanges between systems 610/620 during the course of a chat or game play session.
- Systems 610/620 may comprise an exemplary system like that disclosed in FIGURE 1.
- a system may comprise an image capture device, a video display device, and a client computing device having processor functionality.
- Systems 610/620 would be placed in the context of a user environment, such as a living room or a conference room, which would be located within the field-of-view of the image capture device.
- Systems 610/620 may each be running software for facilitating a video- conference or video 'chaf session or otherwise related to a particular video game involving image data capture.
- the exchange of data between systems 610/620 may be limited to video images and accompanying audio.
- the session may further comprise additional ancillary data such as textual exchanges (e.g., closed captioning); various in-conference applications such as an enterprise application allowing for collaborative development between multiple users (e.g., a word processing document or presentation slide); video-games that may be played on-line during the session; or other multi-media applications (e.g., a media player).
- textual exchanges e.g., closed captioning
- various in-conference applications such as an enterprise application allowing for collaborative development between multiple users (e.g., a word processing document or presentation slide); video-games that may be played on-line during the session; or other multi-media applications (e.g., a media player).
- These data exchanges are in addition to various session protocols related to
- the network adaptor 240 may initiate a conferencing session through, for example, Ethernet connection 242 over the aforementioned network 630.
- Image data (and related audio data) may be captured through the aforementioned image capture device, which may be coupled to the system 610 via, for example, USB interface 220.
- GPU 210 executes various drawing instructions related to not only the actual conferencing data (e.g., user images) but also the inner frame area and outer frame area and related illumination control commands (e.g., 'filling' the outer frame with a particular intensity and color) whereby lighting conditions of a particular environment may be controlled. Instructions related to control of lighting conditions in the particular environment may be a part of the video conferencing software executed by each system 610/620 or at server 640.
- Systems 610/620 may also be configured to manage bandwidth consumption based on certain bandwidth measurements taken by each system 610/620. Bandwidth measurements may also be taken by a third-party application or remote computing device (e.g., server 640) and reported to each system 610/620. Systems 610/620 may adjust certain lighting conditions through screen and frame adjustments in response to locally determinations made with respect to network conditions and bandwidth consumption. Alternatively, systems 610/620 may be configured to receive instructions from a remote computing device such as server 640 or another conference participant's computing device and adjust certain user environment lighting conditions in response to the same.
- a remote computing device such as server 640 or another conference participant's computing device
- address book data may be stored on a removable data storage device as may be accessed through memory card interface 218.
- This 'address book' data may be related to other users of the present video-conferencing/chat system 600.
- the conferencing software may also include certain security features or parental controls to ensure that under-age children do not access the system 600 or are limited to sessions with particular, pre- approved conferees. Such information may also be stored on the aforementioned memory device.
- Conferencing software may further facilitate the generation of video mail such as a 30-second full-motion video. Software may further facilitate entry into various chat-rooms or interaction with certain applications, such as video games.
- Communications network 630 comprises various communications facilities and mediums (e.g., telephony, wireless, satellite, cable, optic, and so forth) as may be provided by telecommunications companies and Internet Service Providers.
- Communications network 630 may be a geographically widespread network (e.g., a Wide Area Network (WAN)), like the Internet that depends upon the aforementioned communications facilities to link various network segments.
- WAN Wide Area Network
- a WAN may ultimately be comprised of smaller linked communications networks such as Local Area Networks (LANs).
- LANs Local Area Networks
- a LAN is typically comprised of a group of computers and other devices dispersed over a relatively limited area and connected by, for example, a variety of broadband communications links.
- LANs may take on a variety of configurations including server client, peer-to-peer, peer groups, or a combination of the same.
- Communications network 630 may also be a closed, proprietary network.
- a conferencing management server 640 may be utilized to conduct various video conferences or other user interactions. For example, system 610 may actually connect with server 640; server 640 will then create a connection with system 620 and a 'chat' session may commence.
- Server 640 may provide various network controls such as connection acceptance, session initiation and termination, bandwidth management, conference recording (e.g., recording the conference session or a transcript of the same for future access); managing chat rooms or group sessions (e.g., sessions involving more than two participants); facilitating certain data exchanges ancillary to the actual conference (e.g., documents, presentations, video games); and also ensuring proper control of lighting conditions in a particular user environment.
- the server 640 may monitor lighting conditions in the conferencing environment of each system (610/620) and adjust certain frame settings through the necessary drawing instructions such that proper lighting conditions are maintained at both ends of the conference. In this regard, server 640 may actually provide the drawing instructions for execution by the appropriate system to 'fill' a frame or adjust the size of the same. Server 640 may also receive various indications from each system (610/620) concerning the quality of lighting conditions at the other system (610/620) and facilitate adjustments as is appropriate by communicating the various indications to various participant devices. [00105] Server 640 may also manage bandwidth consumption in a similar fashion through the control of lighting conditions in various user environments.
- Server 640 may include a bandwidth measurement application for measuring such conditions and responding in an according fashion (i.e., adjusting lighting conditions in a user environment through screen and/or frame adjustments). Server 640 may, alternatively, receive information from each system (610/620) or a third-party application in the communications network 630 concerning various network conditions and bandwidth consumption of the same. Measure of bandwidth conditions may occur over a network 630, at the origin and/or destination of video conference/chat information, at the server 630, as well as at and between various hops in network 630. Server 640 may adjust lighting conditions to allow for bandwidth consumption control in response to a measurement at one or more points in the network 630 as may be deemed appropriate in response to certain measurement information.
- Server 640 may also host the necessary software for the conference to take place.
- the systems 610/620 may comprise only minimal application data necessary to contact the server 640 to indicate the desire to participate in a conference session.
- Critical conferencing architecture software and execution of the same may then occur at the server 640, which may have considerably more processing power than the end-user device (e.g., an entertainment system).
- the more powerful conferencing server like server 640
- conferencing exchanges do not 'slow down' or experience jitter due to processing delays at the various end-users and their related computing devices.
- server 640 may also manage various distributed computing operations.
- server 640 may also manage various distributed computing operations.
- a light emitting diode (not shown) on an image capture device 110 may be used to provide certain strobing conditions or to otherwise operate as an additional artificial light source that may be balanced with the various frame image areas of a display 300.
Abstract
A system and method for controlling lighting conditions in a user environment in the context of an image capture device is disclosed as is a system and method for controlling lighting conditions whereby bandwidth consumption may be managed. An inner frame area and an outer frame area adjust certain lighting conditions in the user environment in response to certain lighting conditions detected by the image capture device; those conditions may affect and be relevant to bandwidth consumption during transmission of the image capture data. The frame areas may be dynamically controlled as to affect the brightness and/or color of the particular user environment.
Description
LIGHTING CONTROL OF A USER ENVIRONMENT VIA A DISPLAY DEVICE
BACKGROUND OF THE INVENTION
Field of the Invention
[0001] The present invention generally relates to the generation of visual data through an image capture device during an audio/visual session such as an
'audio/visual chat' session or during video game play.
[0002] More specifically, the present invention relates to control of lighting conditions in a user environment whereby the bandwidth consumed by transmission of image data generated and/or captured during such audio/visual sessions may be effectively managed. Further, "the present invention relates to control of lighting conditions in a user environment that affect the quality of image data generated during audio/visual sessions.
Description of the Related Art
[0003] With the increased processing capabilities of various computing systems, new methods for interacting with those computer systems have become available. For example, a variety of input devices employing video image capture allow user control of or interaction with objects on a graphical display such as a video monitor.
[0004] Such video input devices often are responsive to the movement or position of a user in the field of view of an image capture device. Video image processing translates the movement of the user that has been captured as a sequence of video images into signals that may be used, for example, to control a character or avatar in a video game. Alternatively, image processing may generate a video image to be displayed in a 'chaf session in a manner similar to a video conference. [0005] An image capture device typically scans a field of view in which a user may be present (e.g., a user environment such as an office, game room, living room or the like). The captured video image may be applied to a video digitizer that provides digital output to a processor that analyzes and processes the digital
information received from the digitizer. Based upon the position or movement of the participant in the field of view, the processor produces signals that are used by the graphics generating system to move objects on the display. Similarly, the system may generate an image of the user for transmission to another user in a chat session. [0006] The output of the devices on graphical displays can be significantly affected by a variety of factors, especially lighting conditions in the user environment. The computer processing time required for image processing in an ideal user environment may be extensive and complex and tends to require substantial computing and/or temporal resources. A user environment that is overexposed or underexposed as to particular light sources only complicates such processing activities in that the system must compensate for the adverse lighting conditions thereby resulting in slower computing operations and affecting any realtime interactions. In some instances, the lighting conditions are so adverse that a computing system cannot compensate for the particular environment conditions and inaccurate data— if any— is generated thereby resulting in incorrect game control or the generation of poor chat session video.
[0007] Similarly, lighting conditions may contribute to the amount of bandwidth consumed by transmission of a captured video image (or images). This may be especially in the context of real-time, full-motion video as may be generated in the context of a video chat session. Video signals, by their very nature, consume large amounts of bandwidth notwithstanding the application of a variety of compression techniques (e.g., lossy compression) as are known in the art. An overwhelming presence of a certain color or colors in a particular environment (e.g., as might be generated by bright lights of a particular color) may be captured by an image capture device. Transmission of that image and its excessive color saturation may increase an overall file size and lend to excessive bandwidth consumption. Excessive bandwidth consumption may, in turn, contribute to a degraded conferencing experience (e.g., jitter).
[0008] There is a need for an image capture device whereby the lighting conditions of various user environments can be automatically and dynamically controlled subject to the particular requirements of the capture device or hardware
and software related thereto. Additionally, there is a need for lessening the computational burdens of an image processing system coupled to such a device whereby the system may function substantially in real-time thus providing the user with a natural interaction experience with regard to a game, chat session, or any other interaction involving the aforementioned image capture device. Further still, there is a need in the art to reduce bandwidth consumption that may occur as a result of the capture of certain image data that increases overall transmission file size.
SUMMARY OF THE INVENTION
[0009] An embodiment of the present invention provides a lighting control system for implementation in a variety of user environments, comprising an image capture device configured to capture image data in the user environment, the image capture device further configured to identify at least one lighting condition in the user environment. A computing device may be configured to generate an inner frame area and an outer frame area on a display device. The inner frame area includes image data captured by the image capture device and the outer frame area controls at least one lighting condition in the user environment. [0010] A further embodiment of the present invention provides a method for controlling lighting conditions in a user environment. The method includes the capture of image data in the user environment and the identification of at least one lighting condition in that environment. An inner frame area and an outer frame area are displayed on a display device whereby at least one lighting condition in the user environment may be controlled through various manipulations of the outer frame area.
[0011] In yet another embodiment of the present invention, a machine- readable medium is provided. An executable program may be embodied in the medium, the program comprising instructions for controlling lighting conditions in the user environment. Those instructions include a method comprising the capture of image data in the user environment; identification of at least one lighting condition in the user environment; and the generation of an inner frame area and an outer frame area on a display device. At least one lighting condition may be controlled in the user environment through various manipulations of the outer frame area.
A still further embodiment of the present invention provides a video conference system. The system comprises a first image capture device configured to capture first user environment image data in a first user environment for transmission over a communications network. A second image capture device may be similarly configured for a similar purpose in a second user environment. A conference
management server may be configured to receive first and second user environment image data and to identify at least one lighting condition in the two environments. Client computing devices associated with the respective image capture devices receive lighting control instructions from the conference management server whereby an outer frame area and an inner frame area are displayed on a display device at each conferencing environment. The outer frame area displayed at each environment may be manipulated such that at least one lighting condition in the respective user environment may be subject to control or modification by the conferencing server.
[0012] In another embodiment, a system for managing bandwidth consumption through control of lighting conditions in a user environment is disclosed. An image capture device captures image data in a user environment, which is then transmitted over a communications network. A computing device may be coupled to the communications network to measure consumption of bandwidth related to transmission of the captured image data over the communications network. A display device may display an inner frame area and an outer frame. The inner frame area and the outer frame area control at least one lighting condition in the user environment in response to the consumption of bandwidth as measured by the computing device.
[0013] In another embodiment of the present invention, a method for managing bandwidth consumption through control of lighting conditions in a user environment is disclosed. Image data is captured and transmitted over a communications network. Consumption of bandwidth related to transmission of the captured image data over the communications network is measured. An inner frame area and an outer frame area may be generated on a display device in the user environment. At least one lighting condition in the user environment may be controlled via the inner frame area and the outer frame area in response to the consumption of bandwidth as measured by the computing device. [0014] A video conference system includes a first image capture device and a second image capture device. Each device is configured to capture user environment image data in a respective user environment and for transmission over a
communications network. A conference management server receives the first user environment image data and the second user environment image data and measures consumption of bandwidth related to transmission of the first user environment image data and the second user environment image data over the communications network. A first user client computing device and a second user client computing device are each associated with a respective image capture device. The first and second user client computing devices receive lighting control instructions from the conference management server in response to the consumption of bandwidth as measured by the conference management server. The lighting control instructions received from the conference management server include generating an outer frame area and an inner frame area on a display device at both the first user client computing device and second user client computing device, the outer frame area and inner frame area collectively controlling at least one lighting condition in the respective user environment.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] FIGURE 1 illustrates an exemplary system for generating real-time, three-dimensional, interactive environment data according to an embodiment of the present invention.
[0016] FIGURE 2 illustrates a block diagram of one embodiment of a client computing device for generating real-time, three-dimensional, interactive environment data.
[0017] FIGURES 3A and 3B illustrate an exemplary video display device comprising an inner frame area and an outer frame area for implementing lighting control of a user environment in accordance with one embodiment of the present invention.
[0018] FIGURES 4A and 4B illustrate an exemplary video display device reflecting different degrees of illumination intensity in the outer frame in accordance with one embodiment of the present invention.
[0019] FIGURE 5A illustrates an exemplary method for implementing lighting control of a user environment through varying illumination intensity in a video display device in accordance with one embodiment of the present invention. [0020] FIGURE 5B illustrates an exemplary method for managing bandwidth with respect to the generation and transmission of image capture data in accordance with one embodiment of the present invention.
[0021] FIGURE 6 illustrates an exemplary network of real-time, three- dimensional, interactive environment data generating systems as may be utilized in a 'chat' session or interactive video game in accordance with one embodiment of the present invention.
DETAILED DESCRIPTION
[0022] FIGURE 1 illustrates an exemplary system 100 for generating realtime, three-dimensional, interactive environment data according to an embodiment of the present invention. The system 100 comprises an image capture device 110, a video display device 120, and a client computing device 130 having processor functionality such as a home entertainment system for video games. The client computing device 130 may be further coupled to a controller device 140 as is shown in FIGURE 1. Generally, a user and a related environment, such as a living room, are located within the field-of-view of the image capture device 110. [0023] The client computing device 130 may be implemented as a
PlayStation® 3 from Sony Computer Entertainment Inc. It should be noted, however, that the client computing device 130 and its processor functionality may be implemented in other types of computing devices such as personal computers, workstations, laptop computers, wireless computing devices, portable media devices, or any other type of computing device that may be capable of receiving and processing graphical image data.
[0024] Video display device 120 may be any device configured for the display of video data such as a television or computer monitor. In one embodiment, video display device 120 receives graphical image data from client computing device 130 via an AV MULTI OUT connector that may be coupled to an integrated audio/video cable and the requisite video and audio connectors. Various other connection mechanisms may also be used, for example, S VIDEO, an RFU adaptor, component video input (Y CB/PB CR/PR), and HDTV input (Y PB PR) as well as the High Definition Multimedia Interface (HDMI) and Digital Visual Interface (DVI). In some embodiments, such as a portable media device, the client computing device 130 and video display device 120 may be integrated as is the case in a PlayStation® Portable from Sony Computer Entertainment Inc.
[0025] Image capture device 110 may be, in one embodiment of the present invention, a color digital camera device with functionality similar to that of a webcam. Image capture device may be coupled to client computing device 130 via a
USB cable. In some embodiments, image capture device 110 may utilize a wireless transmission protocol to exchange data with the client computing device 130 (e.g., 802.1Ix). Image capture device may also include commercially available dedicated video conferencing products such as those manufactured by Polycom, Inc. of Pleasanton, California.
[0026] Image capture device 110 extracts specific information from captured multi-dimensional image data for subsequent presentation to a user of the device (e.g., a color image on visual display device 120 or for transmission to another user over a network as in FIGURE 6 below). For example, captured image data may be integrated into a video game or other user-centric application (e.g., a workout or training video). Captured image data may also be utilized in a video conference or 'audio/visual chat session' wherein the captured data may be transmitted to another user's computing device for display.
10027] Image capture device 110 may be capable of capturing and mapping in three-dimensions in addition to normal two-dimensional video imagery. Similar to normal cameras, image capture device 110 captures two-dimensional data for a plurality of pixels that comprise the video image. These values are color values for each pixel— generally red, green, and blue (RGB) values. In this manner, objects captured by the image capture device 110 appear as two-dimensional objects on a monitor.
[0028] Unlike a conventional camera, however, image capture device 110 may also capture depth values in a particular field-of-view (e.g., a particular user environment). That is, image capture device 110 may capture the x and y components of an environment using RGB values for each pixel in the environment in addition to a z-component, which represents a depth value for the environment. (i.e., the z-axis).
[0029] In operation, a z-value may be captured for each pixel of the scene, each z-value representing a distance from the camera to a particular object in the scene corresponding to the related pixel. A maximum detection range may be defined beyond which depth values will not be detected. Through the use of z-value capture, each object can be tracked in three-dimensions whereby the z-values along
with the two-dimensional pixel data can be processed to create an enhanced three- dimensional interactive environment for the user.
[0030] An input image processor (not shown) at client computing device 130 translates the captured video images and depth data into signals that are delivered to an output image processor (not shown) at the client computing device 130. The output image processor may be programmed to effect movement and status of virtual objects on the video display device 120 in response to signals received from the input image processor. In some embodiments, input and output image processors may be an integrated part of the hardware and software configuration of the image capture device 110. In other embodiments, such processing functionality may be partially distributed over a network like that shown in FIGURE 6, such processing taking place at, for example, a conferencing management server. Some embodiments of the image capture device 110 may also be configured for the capture and subsequent processing of visual data in high-definition.
[0031] Various image processing techniques allow a user to interact with the image capture device 110 device using motion, color detection and, in some embodiments, sound through a built-in microphone or other audio input device (not shown) coupled to the image capture device 110 or client computer device 120. Certain interaction techniques are disclosed in U.S. patent publication number US 2004-0207597 Al for a "Method and Apparatus for Light Input Device" and U.S. patent number 7,102,615 for a "Man-Machine Interface Using a Deformable Device." A prop-input device is disclosed in U.S. patent publication number US 2004-0239670 Al for a "System and Method for Providing a Real-Tϊme Three-Dimensional Interactive Environment" as well as U.S. patent number 6,795,068 for a "Method for Mapping an Object from a Two-Dimensional Camera Image to a Three-Dimensional Space for Controlling Action in a Game Program." The disclosures of all of the aforementioned applications and patents are incorporated herein by reference. [0032] In one embodiment of the present invention, the image capture device liO may be compact in design allowing for placement on top of a video console or television. Image capture device 110 may comprise a pivot thereby allowing for positioning of the device and its related field-of-view. The image capture device 110
may further comprise a rotational ring around the camera lens for manual focus control. Some embodiments of the image capture device 110 may provide for automated positioning and focusing through the use of a directional-sensitive microphone that tracks a source of audio (e.g., an individual who is speaking) and the focus of the device 110 may be subsequently adjusted according to the position of the audio source relative to the image capture device 110.
[0033] Referring now to FIGURE 2, a block diagram of one embodiment of a client computing device 200 (130) is illustrated. Client computing device 200 may be used to aid in the generation of real-time, three-dimensional, interactive environment data. The client 200 may be communicatively coupled to image capture device 110 in order to generate the aforementioned environment data.
[0034] The client 200 may comprise, but is not limited to, a main memory
202, a central processing unit (CPU) 206, vector processing units VUO 204 and VUl 208, a graphics processing unit (GPU) 210, all of which may be coupled via a bus 236 to an input/output processor (IOP) 212. The client 200 may also comprise an IOP memory 214, a controller interface 216, a memory card 218, a Universal Serial Bus (USB) interface 220, and an IEEE 1394 interface 222. The client 200 may further include an operating system read-only memory (OS ROM) 224, a sound processing unit (SPU) 226, an optical, disc control unit 228, and a hard disc drive (HDD) 230, all of which may be connected via a bus 238 to IOP 212.
[0035] Some embodiments of the client 200 may also include a network adaptor 240, which may offer an Ethernet connection 242 and/or telephony connection 244. The client 200, in one embodiment, may be an electronic gaming console although client 200 (or portions thereof) may also be implemented as a general-purpose computer, a set-top box, a hand-held gaming device, or in a mobile device such as a cellular phone. It should further be noted that various other system architectures may be utilized within the scope of the present invention such as the computer architecture and high speed processing model disclosed in U.S. patent publication number 2002-0138637 for a "Computer Architecture and Software Cells for Broadband Networks," the disclosure of which is incorporated herein by reference.
[0036] The CPU 206, the VUO 204, the VUl 208, the GPU 210, and the IOP 212 communicate via a system bus 236. The CPU 206 communicates with the main memory 202 via a dedicated bus 234. The VUl 208 and the GPU 210 may also communicate with one another via a dedicated bus 232. The CPU 206 executes programs stored in the OS ROM 224 and the main memory 202. The main memory 202 may contain pre-stored programs and may also contain programs transferred via the IOP 212 from a CD-ROM, DVD-ROM, or other optical disc (not shown) using the optical disc control unit 228. The IOP 212 controls data exchanges between the CPU 206, the VUO 204, the VUl 208, the GPU 210 and other devices of the system 200, such as the controller interface 216, or from other such systems via the network adaptor 240.
[0037] The GPU 210 executes drawing instructions from the CPU 206 and the
VUO 204 to produce images for display on a display device (not shown). The VUl 208 transforms objects from three-dimensional coordinates to two-dimensional coordinates, and sends the two-dimensional coordinates to the GPU 210. The SPU 226 executes instructions and processes data to produce sound signals that are output on an audio device (not shown).
[0038] A user of the client 200 provides instructions to the CPU 206 via a controller coupled to the controller interface 216. Controller may be any control device, for example, a joystick, a set of directional buttons, and/or other control buttons. An exemplary controller is illustrated in FIGURE 1 (140). A user may instruct the CPU 206 to store certain information on the memory card 218, which may be removable (e.g., a flash memory or other non-volatile memory card), or may instruct a character in a game to perform some specified action. Other devices may be connected to the system 200 via the USB interface 220 and the IEEE 1394 interface 222. As previously noted, the image capture device 110 may be coupled to the client 200 utilizing a USB connector or, in another example, a wireless Ethernet network through Ethernet connection 242.
[0039] In that regard, some embodiments of the client 200 comprise a network adaptor 240 that provides the hardware functionality necessary for the client 200 to connect to a network. The network adaptor 240 may comprise, for
example, a system connector that operates to connect the adaptor 240 to the client computing device 200 through an expansion bus connector 246. The network adaptor 240 may also comprise a power connector and data connector to allow for the provisioning of power from the client 200 to the adaptor 240 and the exchange of data between the client 200 and the adaptor 240. Network adaptor 240 may be fully integrated with the client 200 or may be a detachable hardware device that may be implemented in older legacy client devices 200.
[0040] In some embodiments of the present invention, the network adaptor
240 may also require the installation of certain software on the client 200 to allow for identification and connection to a particular IP address and/or dial-up to a particular Internet Service Provider. Software may also provide other functionalities, such as the creation and maintenance of user profiles, in addition to functional interaction between the client 200 and the network adaptor 240. Such software or data related to such functionality may be embodied on CD-ROMs for games or application requiring a network connection; stored on memory card 218; or part of a firmware upgrade.
[0041] The network adaptor 240 may also comprise an Ethernet connection
242. Through the Ethernet connection 242, a network cable (e.g., a 100 Base-TX or 10- Base T) may be coupled to the network adaptor 240 for connection to a network. The network cable may, for example, be communicatively coupled to a DSL or cable modem. The network cable may also be communicatively coupled to, for example, a router via a LAN port; the router may then be coupled to a DSL or cable modem through a WAN port. In further embodiments, the Ethernet connection 242 may allow for a network cable to be connected to a wireless Ethernet bridge. The wireless Ethernet bridge may be communicatively coupled to a wireless router utilizing, for example, an 802. Hx protocol. The wireless router may be further communicatively coupled to a cable or DSL modem.
[0042] The network adaptor 240 may also comprise a telephony connection
244. Through the telephony connection 244, a standard telephone line with, for example, an RJ-HC telephone connector may be connected to the network adaptor 240 and a telephone wall jack. In this regard, the network adaptor 240 may further
comprise modem functionality such that the client device 200 may communicate data over the public switched telephone network via the telephony connection 244. [0043] FIGURE 3A illustrates an exemplary video display device 300 comprising an inner frame area 310 and an outer frame area 320 for implementing lighting control of a user environment in accordance with one embodiment of the present invention. Lighting control of the user environment may also be used in the context of bandwidth management as further discussed herein. Video display device 300 corresponds to the video display device 120 originally referenced in FIGURE 1 and coupled to client computing device 130 (client 200 of FIGURE 2) for the display of images or other drawing instructions by GPU 210. Video display device 300 may also comprise audio output functionality {e.g., a built-in speaker system) to emit audio signals produced by SPU 226.
[0044] Inner frame area 310 and outer frame area 320 are artificial boundaries that may be created by client computing device 130 as the result of drawing instructions from the CPU 206 being executed by GPU 210. The dimensions of these areas may be determined by or in light of a particular software title being executed (e.g., a video game title utilizing an image capture device 110) or by a server component as might be utilized in an audio/visual 'chaf session as is discussed in further detail in FIGURE 6. In some embodiments— for example, in the case of a 4:3 game shown on a 16:9 display— the outer frame area 320 may be the extra 'dead space' on the sides of a widescreen display.
[0045] Referring to FIGURE 3B, the size of outer frame area 320 is larger than that of the corresponding frame area 320 in FIGURE 3A. Accordingly, the inner frame area 310 of FIGURE 3B is smaller than that of the corresponding frame area 310 in FIGURE 3A. The inner frame 310 and outer frame 320 may also be subject to particular user configurations or settings that may be saved on memory card 218 or some other memory device coupled to the client computing device 130. Inner frame 310 and outer frame 320 may be adjusted in response to certain determinations made by client computing device 130 or in response to certain instructions received from another computing device over a network.
[0046] In one embodiment of the present invention, video game data will be displayed in the inner frame area 310. This could include image data of the user as might be generated by the image capture device 110 or a derived visual game state based upon the image data of the user. For example, the image capture device 110 may capture an image of the user for insertion into a game environment in the form of an avatar. Similarly, the image capture device 110 may generate data used to determine whether the user came into contact with a particular portion of the game environment by 'grabbing' at an object or providing input data through a prop device. This real-world activity or interaction may be reflected in the game environment as a result of the user, for example, swinging at a baseball with a baseball bat prop device in an interactive baseball game. An image capture device 110 tracks and captures the swinging of the baseball bat prop device and, through the necessary images processors, translates that real-world activity into on-screen display data.
[0047] The inner frame area 310 may also be used to display 'chaf session data such as the image of a remote participant communicating with the user. The inner frame area 310 may also display multi-media data generated by the remote participant and transmitted to the present user in the form of a textual 'instant message' or a short video clip. Various types of interactive data may also be displayed in the inner frame area 310 such as an interactive game (e.g., chess or checkers) or a collaborative document (e.g., a report that is being concurrently edited by one or more parties).
[0048] As has been previously noted, image capture device 110 may require certain light levels or light conditions in order for the image capture device 110 to function properly or optimally. If the environment in which the image capture device 110 is operating in lacks sufficient light, certain data (e.g., user interactions or prop input data) may not be detected because the image capture device 110 cannot properly discern various x-y-z position values, color differentiations, or otherwise detect motion of a user or object. Image capture device 110, in conjunction with software and/or hardware operating internal to the image capture device 110, at the client computing device 130, and/or by an intermediate server as is discussed in
FIGURE 6, will determine whether there is proper light present in the environment to properly capture data (e.g., with respect to a target histogram). Similar determinations may be made with regard to, for example, the presence of too much light if the image capture device 110 or a computing device (130) coupled to the image capture device 110 is configured with an infrared (IR) receiver to receive commands from a remote control (e.g., camera on/camera off). Various other spectral emissions may be subject to such user environment light control. [0049] In the context of managing bandwidth consumption as discussed further herein, determinations may be made with respect to the presence of too much light or over-saturation with respect to a particular color. These excesses may result in spikes in bandwidth consumption when transmitting image or video data captured and/or generated by image capture device 110. Spikes in bandwidth consumption may result in a less than optimal conferencing experience, especially in constricted bandwidth environments.
[0050] Image capture device 110 in conjunction with the aforementioned software and/or hardware operating internal to the image capture device 110, at the client computing device 130, and/or by an intermediate server as discussed in FIGURE 6 may determine, for example, that too much red light is present in the environment. This light may come from an external source such as natural sunlight through a window proximate the user environment, from an artificial lighting source in the user environment (e.g., an overhead lamp), or from the display device 300 itself. Utilizing the various display and frame adjustment techniques discussed herein, certain colors may be 'muted/ 'countered' or otherwise 'cancelled' through additive (e.g., mixing of red, green, and blue) and subtractive (e.g., subtracting a hue out of a color scheme by adding more of another) color techniques as are known in color theory.
[0051] In the event that there is a determination that insufficient light exists to properly capture image data or that too much light or a particular hue of light exists and is adversely affecting bandwidth consumption, the outer frame area 320 may be used to provide additional light to the user environment such that the video display device 300 operates as an artificial light source (e.g., certain RGB values of
f individual pixels in the display 300 are adjusted as to average a necessary user environment light level). That is, the outer frame area 320 will become visibly brightened with a particular color and/or intensity of light such that the video display device 300 lights the environment proximate the image capture device 110 whereby image data may be properly captured for subsequent processing. [0052] For example, a user may be utilizing the image capture device 110 to play the aforementioned simulated baseball game with a prop device. As such, the user may find an avatar of themselves inserted into the 'home plate' video game environment and displayed in the inner frame area 310 of FIGURE 3A. If the image capture device 110 and related processing utilities determine that there is insufficient light in the scene environment to properly track the prop device or any other aspect of the user, the video game software operating at client computing device 130 may (via CPU 206) cause the execution of rendering instructions by the GPU 210 that causes the rendering on the display device 300 of a white 'filler' in the outer frame area 320 (i.e., the execution of drawing instructions that causes the individual pixels within the outer frame area 320 to display a particular luminance and/or shading). The inner frame area 310 would continue to display the baseball video game and related information while simultaneously becoming surrounded by a white 'halo' in the outer frame area 320.
[0053] This white 'halo' displayed by the video display device 300 will 'spill' into the real- world environment of the user — that is, the light emitted by the display and measured in candelas per meter squared (cd/m2)— whereby the video display device 300 actually becomes an alternative light source. This additional light generated by the video display device 300 will result in the generation of sufficient additional light to allow for the image capture device 110 to properly capture image data for processing. The intensity of the 'halo' emitted by the outer frame area 320 can be controlled in part by the particular rendering instructions from GPU 210; that is, how white (what shade of white) and how bright (the intensity thereof). [0054] Similarly, if a particular environment exhibits too much of a particular hue of light such that it is degrading transmission of image data (i.e., consuming excess bandwidth), the color and/or intensity of the 'halo' of the outer frame area 320
(and/or the inner frame area 310) may be adjusted and manipulated to compensate for the same. In the context of the Munsell color system, excess saturation (also referred to as chroma) of one particular hue (red) may be countered by a hue of the opposite color (blue/green). The particular saturation / hue combination may be wholly opposite, slightly opposite, or substantially opposite the excess saturation depending on particular network conditions.
[0055] In some embodiments of the present invention, the particular hardware configuration of the display device 300 may be taken into account such that the GPU 210 may optimize the resources at its disposal with regard to manipulating the outer frame area 320 and the light emitted by the same. These configurations may be automatically detected by the client computing device 200 or provided to the device 200 through user input as to particular configurations, the brand and model of the display device (which in turn corresponds to particular operating specifications), and so forth. In some embodiments, device 200 may consult a look-up table or other data store (either locally or over a network) to identify particular operating capabilities (e.g., bit depth) for a particular display model if those capabilities are not immediately identified by the user or through certain plug-and-play (PnP) functionality.
[0056] For example, a liquid crystal display (LCD) in a flat screen television comprises a certain resolution (i.e., the number of dots of color — pixels — on the display) referenced by a horizontal axis (rows) and vertical axis (columns). A wide aspect 22-inch LCD monitor may conform to the WUXGA (Wide Ultra Extended Graphics Array) standard and comprise a resolution of 1920x1200 whereas a smaller device such as a 15-inch LCD monitor may only conform to the XGA (Extended Graphics Array) standard and comprise a resolution of 1024x768. While an LCD display device is referenced, the present invention is equally as applicable to a CRT- based display.
[0057] Additionally, the particular display mode of a video display device
300 determines how many colors that device 300 can display. For example, a display that is configured to operate in SuperVGA mode can display 16,777,216 colors (also known as true color) the result of a 24-bit-long description of a pixel (the bit depth).
For example, in 24-bit bit depth, eight bits are dedicated to each of the three additive primary colors (RGB). The bit depth therefore determines the number of colors that can be displayed at one time. To create a single colored pixel an LCD, for example, uses three sub-pixels with red, green, ad blue filers. Subject to the control and variation of the voltage applied, the intensity of each sub-pixel can range over 256 shades. Combining the sub-pixels produces a possible palette of 16.8 million colors (256 shades of red x 256 shades of green x 256 shades of blue).
[0058] The brightness of the light emitted into the user environment can also be affected by the size of the outer frame area 320 relative the inner frame area 310. Returning to FIGURE 3A, by filling the outer frame area 320 with white image data, a certain degree of luminance may be created. Identical white image data (i.e., the same intensity and shade) in FIGURE 3B, however, will create a greater degree of luminance in that the outer frame area 320 of FIGURE 3B is larger than that of FIGURE 3A. Thus, the amount of light that is generated by the video display device 300 though a 'halo' may be dynamically controlled during the use of image capture device 110.
[0059] For example, the image capture device 110 and related client device
130 may be in a room with a window in the late afternoon. While there may be sufficient natural light to support image data capture at the time game play commences, as the afternoon continues and the natural light source begins to disappear (i.e., the sun sets), there may be insufficient light for the image capture device 110 to properly continue to capture image data. The outer frame area 320 may, at that point, be illuminated as a white halo to provide the additional light necessary to allow for game play to continue. The intensity of the light, as noted, may be controlled by the client computing device 130 that may be running game software that has been calibrated with the image capture device 110 to determine when the proper amount of light is present to allow for the game to function as intended. In that regard, requisite light settings may be predetermined or calibrated based on the particular game and/or user environment.
[0060] As natural light continues to disappear throughout the course of the day, the intensity of the artificial light source emitted from the display device 300
may be gradually increased. For example, a dull gray light source in the outer frame area 320 may provide sufficient light at 4.00 PM but a flat white emission may be required from the outer frame area 320 at 6.00 PM. The intensity of the light may also be increased (e.g., the brightness of the light). At some point, the limitations of the video display device 300 may be such that even the most intense white available does not provide sufficient 'artificial lighf to allow game play to continue. At that point, the GPU 210 may cause for the size of the inner frame area 310 to decrease and the outer frame area 320 to increase thereby providing additional artificial light (i.e., additional individual pixels in the screen display utilized for illumination purposes versus the display of game data). The dynamic adjustments described above and in the context of maintaining proper lighting conditions in a user environment are equally applicable in the context of bandwidth management.
[0061] In some instances, particular portions of the user environment may be subject to insufficient lighting conditions. For example, the right side of a room may have an open window whereby sufficient light may be provided to allow for image data capture. The left side of the room, however, may have window shades drawn such that no natural light may be entering that portion of the user environment. The orientation of the inner frame area 310 and outer frame 320 may be adjusted such that the inner frame area 310 may be moved to the far right hand side of the display 300 whereby the inner frame area 310 may be off-center but providing a larger, focused outer frame area 320 on the left-side of the display such that additional artificial light may be generated by the display 300. The positioning and centering of the frame areas 310 and 320 may be dynamically adjusted as user of the image capture device 110 continues.
[0062] Alternatively, particular gradations of color from one-side of the display 300 to another may be implemented wherein the orientation of the inner frame area 310 and the outer frame area 320 may remain constant. In the aforementioned example where the left side of a room is darker than the right, the portion of the outer frame area 320 relative the left side of the room may emit a white light while the right side of the outer frame area 320 displays a neutral background color in that sufficient natural light is present on the right side of the room. A
gradual gradation of color (from white to black) between the left side of the outer frame 320 to the right side may bridge the two 'sides' of the outer frame area 320. [0063] Particular portions of the outer frame area 320 (or other frame areas in the case of multiple frames surrounding the inner frame area 310) may be subdivided into various sub-sections. Each of those sub-sections may be subject to various lighting controls. For example, the outer frame area 320 may be sub-divided into four quadrants: an upper right, upper left, lower right, and lower left quadrant. Each of those individual quadrants may be subject to various lighting controls (e.g., the upper right and lower left corner of the frame may be illuminated while the upper left and lower right are not). These various controls include color, luminance, and any other condition that may be controlled by the various systems and methods disclosed herein.
[0064] The various sub-divisions may also be subjected to various shapes for both functional and aesthetic reasons. .For example, instead of squares and rectangles, various triangle or other polygonal configurations may be implemented. Various circular or elliptical patterns may also be used. These and any other shape or design capable of being displayed (e.g., lines, squiggles, waves, dots, splotches, etc.) may be uniformly or randomly displayed as a group or with a variety of shapes and each object subject to lighting control. Various shapes and patterns may also be subject to strobing (i.e., regular, controllable series of high power flashes rather than continuous light) for a variety of effects.
[0065] For example, strobing may be used to create a strobe light feature in the user environment or, similarly, to counter an effect caused by natural or other artificial light conditions in the user environment. Certain video games or television programs that may cause epileptic fits, feelings of vertigo, and the like and that might be displayed in the inner frame area 310 may be countered through a counter- strobe effect in the outer frame area 320. In that regard, the presently described lighting controls systems and methods may also be implemented with traditional television programming in addition to 'chat' sessions and video games. [0066] As noted, certain calibrations may take place through the image capture device 110. The image capture device 110 may, at start-up, sample the user
environment to determine various lighting conditions and the effect those conditions will have on image capture over a particular operating range (e.g., lowest possible range of capture to an optimal range of capture). The image capture device, in conjunction with the various lighting controls discussed herein as implemented by various hardware and/or software operations, may then adjust certain user environment lighting conditions through inner frame area 310 and outer frame area 320 lighting manipulations (e.g., size, brightness, orientation, color). During the course of these adjustments, the image capture device 110 may 're-sample' the environment to determine what environmental control condition will allow for a particular operation of a game or chat session in light of the particular environment. [0067] The computing device 130 in conjunction with particular 'chaf or video game software may make that determination or such a determination may come as a result of a management server determination as is described in the context of FIGURE 6. Similarly, the determination of a particular user environment condition may come from a user in another environment if that particular user is unable to properly view the first user in a chat session. That decision by a second user may be made by selecting an available environment setting in light of several environmental control conditions.
[0068] For example, during the start-up calibration, the first user environment may be subject to three different intensities of a particular color of light; three different colors of light; and three different frame area sizes. The image capture device 110 may capture an image of each of those nine possible environments. The effect of that particular environment control may then be displayed to the second user in the form of a still frame or even a short (e.g., 5 second) video clip. The second user may then select the particular environment control based on the image or clip that appears best. A user operating in a solo environment (e.g., playing a video game) may select a particular environmental control in a similar fashion. [0069] These calibrations or adjustments may occur at start-up of a game or chat session or may occur dynamically during the session. For example, adjustments may occur automatically during game play or at breaks such as between levels. Adjustments may further occur in response to a user action wherein a query may be
made such as during a pause. In some embodiments, a particular frame of data (e.g., an I-frame) may indicate the propriety and need to readjust lighting conditions in the user environment.
[0070] Similarly, particular frames of data may be recognized during the decoding process as likely to cause certain changes in the user environment. For example, incoming image data of a conference remote participant may consist of a bright white background that will spill into the user environment thereby causing an unintended change in user environment lighting conditions. These environment changing conditions may be identified in advance such that when the actual image data may be rendered that display 300 will have already adjusted an inner frame area 310 and/or outer frame area 320 as may be appropriate such that no adverse affect or change to the user environment occurs.
[0071] It should be noted that in some embodiments of the present invention, the outer frame area 320 may always be present but simply unutilized (i.e., a neutral background color such as black). The frame area may then 'filled' as needed in response to GPU 210 executing a drawing instruction from the CPU 206. Ih other embodiments, the actual graphic images to be displayed may occupy the majority of the game screen and, as such, only the inner frame area 310 may be displayed. If and when an artificial light emission is necessary from a 'halo' generated by the outer frame area 320, the software executed by the CPU 206 and otherwise controlling the display of the frame areas 310 and 320 and interacting with the image capture device 110 may cause the GPU 210 to reconfigure the game environment (e.g., re-size the game environment to fit in a small inner frame area 310) such that the outer frame area 320 may now be displayed and emit the appropriate 'halo' of artificial light. [0072] The re-sizing of the various frame areas may occur during the course of game play (e.g., the re-sizing occurs as game play progresses). Alternatively, the re-sizing may occur only during pauses in game action such as to avoid the size of the environment and various characters changing while game play is occurring (i.e., to avoid a vertigo-type effect). Additionally, re-sizing may actually cause an automatic pause in game play with a query issued to the user as to whether re-sizing is permitted. In these instances, the amount of natural light may be sufficient to
allow game play to continue but not without some processing errors. If game play deteriorates to the point that additional light is necessary, the user may be asked whether they wish to activate the illumination control feature of the present invention. If not (e.g., the user does not wish to minimize the size of the actual game environment in the inner frame area 310), the user may manually provide another source of artificial light through, for example, an overhead lamp or other light source in the actual user environment. Various user input may be provided through, for example, controller 140 in addition to other inputs such as voice-recognition. [0073] In some embodiments, different colors or shades of colors may also be utilized to maximize the color balance of a particular user environment. For example, a user may have different light sources offering different colors of light. These varying colors may cause difficulty with the image capture device 110 processing environment information, especially with regard to color differentiation. In these instances, the image capture device 110 may identify irregularities or processing difficulties in the environment in a manner similar to the identification of a lack of sufficient light. The outer frame area 320 may used in a similar manner as to produce different colors and intensity of light to counterbalance certain overexposures in the particular environment. In some embodiments, additional frame areas may be used (e.g., a third, fourth, or fifth frame area) such that combinations of certain colors (e.g., yellow and blue individual halos to create an overall green appearance) may be implemented.
[0074] FIGURES 4A and 4B illustrate an exemplary video display device 400 reflecting different degrees of illumination intensity in the outer frame area 420 in accordance with one embodiment of the present invention. FIGURES 4A and 4B, which reflect a 'video chaf session, illustrates the lighting adjustments of a user environment described in the context of FIGURES 3A and 3B. In that regard, the inner frame area 410 of FIGURE 4A reflects image data related to another participant (i.e., a remote chat participant); the outer frame area 420 is presently in a neutral background state.
[0075] In FIGURE 4B, the outer frame area 420 has been 'filled' such that it now reflects white light 'halo' as referenced in the context of FIGURES 3A and 3B.
The inner frame area 410 has also been reduced in that the lighting conditions in the environment of the user receiving the present image have made it necessary to increase the amount of light emitted into that environment. As such, the outer frame area 420 was expanded at the expense of the inner frame area 410. [0076] In some embodiments of the present invention, the inner frame area
410 may need to be reduced in that the graphic data in the inner frame area 410 is inadvertently creating excess light and thereby over saturating the user environment. For example, in the aforementioned baseball game example, if the user hits a home run, the image displayed may pan upward to the sky to follow the path of the baseball as it travels toward the outskirts of the ballpark. As the scene pans toward the sky, the screen display will become predominantly white and gray (i.e., colors of the sky and clouds). These bright colors may 'spill' into the user environment, which may cause image data capture difficulties. In response to such a situation, the inner frame area 410 may decrease in size to reduce the amount of secondary light 'spillage' into the user environment. Additionally, the outer frame area 420 may be darkened to help counter suddenly excess bright light being emitted from the inner frame area 410.
[0077] FIGURE 5A illustrates an exemplary method 500 for implementing lighting control of a user environment through varying illumination intensity in a video display device in accordance with one embodiment of the present invention. [0078] In step 510 an image capture device may attempt to capture image data from a user environment. In step 520, a determination may be made whether adverse lighting conditions are inhibiting the processing of the image capture data or the actual capture of the data. If there are no adverse conditions present and the image data is captured and processed without incident, image data capture proceeds as necessary followed by subsequent processing of the data.
[0079] If it is determined in step 520 that adverse lighting conditions do exist, an attempt may be made to adjust display intensity in step 530. This may occur through various adjustments whereby the outer frame area may be 'filled' with a particular color and intensity of light as discussed in the context of FIGURES 3A and 3B and 4A and 4B above. If the adjustment as to display intensity is not sufficient,
then (in step 540), frame image boundaries may be enlarged (or reduced as is appropriate) in order to increase or decrease the total amount of artificial light projected into the user environment.
[0080] While step 540 and the adjustment of frame boundaries is recited as following step 530 and the adjustment of light intensity, this is not to suggest the necessity of a step-by-step process. Both adjustments may occur concurrently or frame adjustment may occur prior to intensity adjustment in that FIGURE 5A is an exemplary embodiment only. Regardless of the order, following the adjustment of image intensity and/or frame boundaries, further attempts at image capture and/or image data processing occur as the method repeats or continues (as is appropriate) in step 510.
[0081] FIGURE 5B illustrates an exemplary method 550 for managing bandwidth with respect to the generation and transmission of image capture data in accordance with one embodiment of the present invention. Specifically, FIGURE 5B illustrates a method 550 for managing bandwidth through the implementation of lighting control of a user environment through, for example, varying illumination intensity in a video display device.
[0082] As discussed in U.S. patent application number 11/624,886, the control of lighting conditions in a user environment may be associated with bandwidth and processing availability. In MPEG compression, images are represented in YUV color space (YCbCr) wherein 24-bits per pixel are present: 8-bits for luminance (Y) and 8- bits for each of the two chrominance (U), (V) elements. The chrominance information in the YUV color space data may be sub-sampled in both the horizontal and vertical direction. All of the luminance information may be retained, however, as the human eye is more sensitive to luminance information rather than chrominance information. Frames of video are subsequently divided into 16x16 macro-blocks consisting of four 8x8 luminance blocks and two 8x8 chrominance blocks (1 U and 1 V).
[0083] Each frame of video data may then be encoded as one of three types of frames: intia-frame (I-frames), forward predicted frame (P-frames), and bidirectional predicted frames (B-frames). In the case of an I-frame, the frame may be
encoded as a single image with no reference to past or future frames. Further, with the exception of data quantization following the encoding of each 8x8 block from a spatial domain to a frequency domain utilizing the Discrete Cosine Transform (DCT) and the aforementioned sub-sampling, there is no lossy compression in an I-frame — especially with regard to luminosity. The other frame types — P-frames (relative a past reference frame) and B-frames (relative a past reference frame, a future reference frame or both)— refer to the I-frame. Thus, decreasing certain luminosity information in the I-frame would represent savings with regard to the all of the frames in a stream of video data.
[0084] In this regard, the image data captured from the user environment could be accepted at the lowest possible operating condition. That is, lighting controls would illuminate a user environment no more so than necessary in order to reduce the amount of luminosity data being captured, compressed and subsequently transmitted over a network. This decreased luminosity data represents not only savings in bandwidth at a transmitting computing device, a receiving computing device, and over the network backbone but can also recognize a savings with regard to processing cycles at a computing device perform digitization and compression following image capture.
[0085] Similarly, certain colors captured from a particular user environment may be captured and/or cancelled out by adjusting display emissions and/or frame boundaries as discussed above. For example, in the context of a YUV 4:4:4 environment— where Y pertains to luminance (brightness) and U and V are chrominance (color) components — each channel is expressed with 8 bits (256 possible values) and uses the same amount of space. As noted above, the human eye is most sensitive to luminance— the black and white information in picture detail. Less sensitive areas of the human eye "fill in" color. That is, color information may be reduced without the human eye noticing the same. Thus, the chrominance components (U) and (V) may be reduced without the human eye adversely noticing the same. Bandwidth consumption may be reduced by adjusting chrominance information.
17
[0086] For example, the environment may be effectively captured at (or manipulated to conform to) the equivalent of 4:2:2 format whereby the chroma components are sampled at half the rate of luma thereby reducing bandwidth of the signal by as much as one-third with little no perceptible visual difference. Specifically, red and blue channels of the video signal may be half the bandwidth of the luminance information. For example, green may be cancelled out as green can be calculated from red and blue chrominance information in the context of luminance information.
[0087] A utility related to preservation of bandwidth and processing power vis-a-vis the image capture device and/or related hardware or software may be manually implemented by a user. For example, a user may recognize that their particular computing device lacks accelerated processing power or may be coupled to a low bandwidth network or, similarly, that a recipient of data captured by the image capture device suffers from similar limitations. In these instances, the user may activate such a utility to avoid bottlenecks of data.
[0088] Similarly, this preservation utility may be implemented on an as- needed basis, the need being recognized by a network congestion monitoring utility like ping, which utilizes the Internet Control Message Protocol (ICMP), and trace route, which utilizes the Uniform Datagram Protocol (UDP), to measure network response time. These and other network utilities may be implemented in various network devices {e.g., switches or routers) or through large-scale platforms like the Network Application Performance Analysis Solution (NAPAS) for Cisco Systems. Indicia of network congestion generated by these applications and solutions may be provided to a computing device operating certain software related to image capture (e.g., a client computing device or a conference management server) such that bandwidth and/or processor preservation may be activated through the manipulation of lighting conditions that are not optimal but otherwise sufficient to allow for continued image data capture.
[0089] In certain embodiments wherein one end of a video data exchange degrades a particular lighting condition for the purposes of improved processing or transmission time, the recipient of that data may adjust the incoming video data such
that it does not appear as lesser quality image data. For example, the incoming video data may be pre-processed such that luminance that was degraded by the sender of the data may be artificially amplified by the recipient through editing or video preprocessing editing software that automatically adjusts the data as it is decoded and rendered.
[0090] In step 560, the bandwidth being consumed by image capture data transmitted over a network (as may occur in the context of a video chat session like that described in FIGURE 6 below) is made. A determination of bandwidth may be made utilizing, for example, the aforementioned NAPAS or simpler methodologies such as ping and trace route. The determination may occur at the transmitting computing device, at the receiving computing device, or at some intermediate computing device or at some other point in a communications network. The information may then be provided to the transmitting device such that it may adjust its image capture operations as is appropriate and is discussed in the context of steps 580 and 590. Alternatively, another device may make certain determination as to bandwidth consumption and provide instructions to the transmitting computing device such that it may responsively adjust its image capture operations as may occur through screen and/or frame adjustment.
[0091] In step 570, a determination is made as to whether image capture data that is being transmitted over a network exceeds a particular bandwidth limitation. The particular bandwidth may be a static predetermined amount (e.g., a target threshold). That static threshold may be manually set by a user or network administrator. The static threshold may also be identified at the outset of a chat session and remain constant throughout the session. Alternatively, the amount may be determined in the context of a particular video chat session and may change during the course of the chat session. That is, as network conditions change from the beginning to the end of the session and at various points in between, the target threshold may be adjusted subject to those changing network conditions. [0092] In one example, two users chatting over a dedicated corporate network for such communications may be unlikely to encounter other users or applications attempting to utilize the same finite amount of bandwidth in the
network. As such, even if captured image data consumes large amounts of bandwidth, the availability of large amounts of network overhead may obviate the need to otherwise adjust network traffic consumption. On the other hand, if two users are chatting over a public network shared by other users at a peak traffic time, even image capture data that is 'normal' under the circumstances may exceed a particular bandwidth allotment or may otherwise encounter transmission difficulties due to other users and applications sharing the network. Further, if a chat session commences amongst a group of fifteen participants, bandwidth consumption may initially be high with respect to the group and available resources. As participants drop out of the conference (e.g., their presence is no longer required), the total bandwidth consumption may decease and what constitutes an excessive use of bandwidth for any one user may change as additional resources become available to the session.
[0093] If a determination is made that image capture data does not exceed a particular bandwidth (either predetermined or determined in a dynamic fashion), then measurement continues in step 560. Measurement of bandwidth consumption may occur at predetermined times during the chat session (e.g., every five minutes). Measurement may also occur upon the occurrence of a particular event, for example, transmission of the I-frame or every nlh I-frame. When and how often measurement occurs may also be set by the user or a network administrator.
[0094] If it is determined that image capture data being transmitted over the network exceeds a particular bandwidth limitation, adjustment of display intensity and frame boundaries may occur in steps 580 and 590, These particular adjustments may be similar with respect to the display and frame adjustments that occurred in steps 530 and 540 in the context of FIGURE 5A. Like FIGURE 5A, adjustment of display intensity and frame boundaries are not necessarily a step-by-step process. Both adjustments may occur concurrently or frame adjustment may occur prior to intensity adjustment. Further, only one particular adjustment (i.e., frame but not display) may occur.
[0095] Lighting adjustment sources other than or in addition to the screen and a frame area within the screen may be utilized in the context of managing
bandwidth consumed by image capture data. For example, an external light source may be coupled to a device controlling lighting emissions (e.g., client computing device 130 of FIGURE 1). This coupling may occur in the context of a USB connection or other data connection. The external lighting device may be proximate or otherwise remote from the actual image capture device and related computing device so long as the light source may be configured to result in some influence of the lighting conditions of a particular user environment. Some lighting control hardware may be integrated with particular display device such as a backlight or other ambient light projector.
[0096] FIGURE 6 illustrates an exemplary network 600 of real-time, three- dimensional, interactive environment data generating systems 610/620 as may be utilized in a 'video chat' session or interactive video game and in accordance with one embodiment of the present invention. Systems 610/620 exchange data over communications network 630. Optional conferencing management server 640 may facilitate exchanges between systems 610/620 during the course of a chat or game play session.
[0097] Systems 610/620 may comprise an exemplary system like that disclosed in FIGURE 1. In that regard, such a system may comprise an image capture device, a video display device, and a client computing device having processor functionality. Systems 610/620 would be placed in the context of a user environment, such as a living room or a conference room, which would be located within the field-of-view of the image capture device.
[0098] Systems 610/620 may each be running software for facilitating a video- conference or video 'chaf session or otherwise related to a particular video game involving image data capture. The exchange of data between systems 610/620 may be limited to video images and accompanying audio. The session may further comprise additional ancillary data such as textual exchanges (e.g., closed captioning); various in-conference applications such as an enterprise application allowing for collaborative development between multiple users (e.g., a word processing document or presentation slide); video-games that may be played on-line during the session; or other multi-media applications (e.g., a media player). These data exchanges are in
addition to various session protocols related to establishing and maintaining the exchange of conferencing and other related data.
[0099] Through the aforementioned software, which may be accessed through optical disk control unit 228, the network adaptor 240 may initiate a conferencing session through, for example, Ethernet connection 242 over the aforementioned network 630. Image data (and related audio data) may be captured through the aforementioned image capture device, which may be coupled to the system 610 via, for example, USB interface 220. GPU 210, in turn, executes various drawing instructions related to not only the actual conferencing data (e.g., user images) but also the inner frame area and outer frame area and related illumination control commands (e.g., 'filling' the outer frame with a particular intensity and color) whereby lighting conditions of a particular environment may be controlled. Instructions related to control of lighting conditions in the particular environment may be a part of the video conferencing software executed by each system 610/620 or at server 640.
[00100] Systems 610/620 may also be configured to manage bandwidth consumption based on certain bandwidth measurements taken by each system 610/620. Bandwidth measurements may also be taken by a third-party application or remote computing device (e.g., server 640) and reported to each system 610/620. Systems 610/620 may adjust certain lighting conditions through screen and frame adjustments in response to locally determinations made with respect to network conditions and bandwidth consumption. Alternatively, systems 610/620 may be configured to receive instructions from a remote computing device such as server 640 or another conference participant's computing device and adjust certain user environment lighting conditions in response to the same.
[00101] In some embodiments of the present invention, address book data may be stored on a removable data storage device as may be accessed through memory card interface 218. This 'address book' data may be related to other users of the present video-conferencing/chat system 600. The conferencing software may also include certain security features or parental controls to ensure that under-age children do not access the system 600 or are limited to sessions with particular, pre-
approved conferees. Such information may also be stored on the aforementioned memory device. Conferencing software may further facilitate the generation of video mail such as a 30-second full-motion video. Software may further facilitate entry into various chat-rooms or interaction with certain applications, such as video games. [00102] Communications network 630 comprises various communications facilities and mediums (e.g., telephony, wireless, satellite, cable, optic, and so forth) as may be provided by telecommunications companies and Internet Service Providers. Communications network 630 may be a geographically widespread network (e.g., a Wide Area Network (WAN)), like the Internet that depends upon the aforementioned communications facilities to link various network segments. In that regard, a WAN may ultimately be comprised of smaller linked communications networks such as Local Area Networks (LANs). A LAN is typically comprised of a group of computers and other devices dispersed over a relatively limited area and connected by, for example, a variety of broadband communications links. LANs may take on a variety of configurations including server client, peer-to-peer, peer groups, or a combination of the same. Communications network 630 may also be a closed, proprietary network.
[00103] In some embodiments of the present invention, a conferencing management server 640 may be utilized to conduct various video conferences or other user interactions. For example, system 610 may actually connect with server 640; server 640 will then create a connection with system 620 and a 'chat' session may commence. Server 640 may provide various network controls such as connection acceptance, session initiation and termination, bandwidth management, conference recording (e.g., recording the conference session or a transcript of the same for future access); managing chat rooms or group sessions (e.g., sessions involving more than two participants); facilitating certain data exchanges ancillary to the actual conference (e.g., documents, presentations, video games); and also ensuring proper control of lighting conditions in a particular user environment. [00104] In the latter example, the server 640 may monitor lighting conditions in the conferencing environment of each system (610/620) and adjust certain frame settings through the necessary drawing instructions such that proper lighting
conditions are maintained at both ends of the conference. In this regard, server 640 may actually provide the drawing instructions for execution by the appropriate system to 'fill' a frame or adjust the size of the same. Server 640 may also receive various indications from each system (610/620) concerning the quality of lighting conditions at the other system (610/620) and facilitate adjustments as is appropriate by communicating the various indications to various participant devices. [00105] Server 640 may also manage bandwidth consumption in a similar fashion through the control of lighting conditions in various user environments. Server 640 may include a bandwidth measurement application for measuring such conditions and responding in an according fashion (i.e., adjusting lighting conditions in a user environment through screen and/or frame adjustments). Server 640 may, alternatively, receive information from each system (610/620) or a third-party application in the communications network 630 concerning various network conditions and bandwidth consumption of the same. Measure of bandwidth conditions may occur over a network 630, at the origin and/or destination of video conference/chat information, at the server 630, as well as at and between various hops in network 630. Server 640 may adjust lighting conditions to allow for bandwidth consumption control in response to a measurement at one or more points in the network 630 as may be deemed appropriate in response to certain measurement information.
[00106] Server 640 may also host the necessary software for the conference to take place. In such an instance, the systems 610/620 may comprise only minimal application data necessary to contact the server 640 to indicate the desire to participate in a conference session. Critical conferencing architecture software and execution of the same may then occur at the server 640, which may have considerably more processing power than the end-user device (e.g., an entertainment system). By leaving heavy processing to the more powerful conferencing server (like server 640), conferencing exchanges do not 'slow down' or experience jitter due to processing delays at the various end-users and their related computing devices. In some embodiments of the present invention, like those utilizing a high speed processing model as disclosed in U.S. patent publication number 2002-0138637 for a
"Computer Architecture and Software Cells for Broadband Networks," server 640 may also manage various distributed computing operations. [00107] While the present invention has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the true spirit and scope of the present invention. In addition, modifications may be made without departing from the essential teachings of the present invention. For example, a light emitting diode (not shown) on an image capture device 110 may be used to provide certain strobing conditions or to otherwise operate as an additional artificial light source that may be balanced with the various frame image areas of a display 300.
[00108] Various alternative systems may be utilized to implement the various methodologies described herein and various methods may be used to achieve certain results from the aforementioned systems. Certain methodologies and benefits discussed in the context of lighting control of a user environment for detection of user interactions may be equally as applicable to controlling bandwidth consumption with respect to captured and transmitted image data and vice versa. [00109] The steps of the processes disclosed herein and various alternatives of the same may be embodied in hardware or software. Such embodiments may include a computer-readable medium such as an optical disc, memory card, carrier waves, and the like. Such mediums may include instructions executable by the processor of a computing device.
Claims
1. A system for controlling lighting conditions in a user environment, comprising: an image capture device configured to capture image data in a user environment, the image capture device further configured to identify at least one lighting condition in the user environment; a computing device configured to generate an inner frame area and an outer frame area; and a display device configured to display the inner frame area and the outer frame area, wherein the inner frame area includes image data captured by the image capture device and the outer frame area controls at least one lighting condition in the user environment.
2. The system of claim 1, wherein the computing device is further configured to adjust the size of the inner frame relative to the outer frame.
3. The system of claim 1, wherein the captured image data is integrated with a video game.
4. The system of claim 1, wherein the captured image data is transmitted to a remote computing device.
5. The system of claim 4, wherein the remote computing device is a conferencing management server.
6. The system of claim 4, wherein the remote computing device is associated with a video conference participant.
7. The system of claim 1, wherein the image capture device is further configured to track an object in the user environment in response to a source of audio tracked by a directional-sensitive microphone.
8. The system of claim 7, wherein the image capture device automatically focuses on the object in accordance with the position of the audio source relative to the image capture device.
9. The system of claim 1, wherein the image capture device includes a light emitting diode (LED) configured to create a strobing condition in the user environment.
10. A method for controlling lighting conditions in a user environment, comprising: capturing image data in the user environment; identifying at least one lighting condition in the user environment; generating an inner frame area and an outer frame area on a display device; and controlling at least one lighting condition in the user environment through the outer frame area.
11. The method of claim 10, wherein controlling the at least one lighting condition includes increasing the brightness of the user environment.
12. The method of claim 10, wherein controlling the at least one lighting condition includes adjusting the color of the user environment.
13. The method of claim 11, wherein the brightness of the user environment is controlled through the intensity of the outer frame area.
14. The method of claim 11, wherein the brightness of the user environment is controlled through the size of the outer frame area.
15. The method of claim 11, wherein the brightness of the user environment is controlled through a combination of intensity and size of the outer frame area.
16. The method of claim 15, wherein the intensity and size are dynamically controlled as the at least one lighting condition in the user environment changes.
17. A computer-readable storage medium having embodied thereon a program executable by a processor, the program comprising instructions for controlling lighting conditions in a user environment, the method comprising: capturing image data in the user environment; identifying at least one lighting condition in the user environment; generating an inner frame area and an outer frame area on a display device; and controlling at least one lighting condition in the user environment through the outer frame area.
18. The computer-readable storage medium of claim 17, further comprising controlling the at least one lighting condition through a change in the brightness of the user environment.
19. The computer-readable storage medium of claim 17, further comprising controlling the at least one lighting condition through a change in the color of the user environment.
20. The computer-readable storage medium of claim 17, wherein the brightness of the user environment is controlled through the intensity of the outer frame area.
21. The computer-readable storage medium of claim 17, wherein the brightness of the user environment is controlled through the size of the outer frame area.
22. The computer-readable storage medium of claim 18, wherein the brightness of the user environment is controlled through a combination of intensity and size of the outer frame area.
23. The computer-readable storage medium of 22, wherein the intensity and size are dynamically controlled as the at least one lighting condition in the user environment changes.
24. A video conference system, comprising: a first image capture device configured to capture first user environment image data in a first user environment for transmission over a communications network; a second image capture device configured to capture second user environment image data in a second user environment; for transmission over the communications network; a conference management server coupled to the network, the conference management server configured to receive the first user environment image data and the second user environment image data, the conference management server further configured to identify at least one lighting condition in the first user environment and the second user environment; a first user client computing device associated with the first image capture device, the first user client computing device configured to receive lighting control instructions from the conference management server; and a second user computing device associated with the second image capture device, the second user client computing device configured to receive lighting control instructions from the conference management server, wherein the lighting control instructions from the conference management server comprise generating an outer frame area and an inner frame area, the outer frame area controlling at least one lighting condition in the respective user environment.
25. The video conference system of claim 24, wherein the conference management server is further configured to exchange data ancillary to the video conference with the first user client computing device and the second user client computing device.
26. The video conference system of claim 25, wherein the ancillary data comprises application data.
27. The video conference system of claim 25, wherein the ancillary data comprises text data.
28. The video conference system of claim 25, wherein the ancillary data comprises multi-media data that is not the actual conferencing data.
29. The video conference system of claim 25, wherein the ancillary data comprises a video-game.
30. The video conference server of claim 24, wherein the conference server is further configured to record image capture data exchanged between the first client computing device and the second client computing device.
31. A system for managing bandwidth consumption through control of lighting conditions in a user environment, comprising: an image capture device configured to capture image data in a user environment; a computing device coupled to a communications network, the computing device configured to measure consumption of bandwidth related to transmission of the captured image data over the communications network; a display device configured to display an inner frame area and an outer frame area, wherein the inner frame area and the outer frame area control at least one lighting condition in the user environment in response to the consumption of bandwidth as measured by the computing device.
32. The system of claim 31, wherein the size of the inner frame may be adjusted relative to the size of the outer frame.
33. The system of claim 31, wherein the captured image data is transmitted to a remote computing device as a part of a video conference.
34. The system of claim 33, wherein the remote computing device is a conferencing management server.
35. The system of claim 33, wherein the remote computing device is associated with a video conference participant.
36. The system of claim 31, wherein the image capture device is further configured to track an object in the user environment in response to a source of audio tracked by a directional-sensitive microphone.
37. The system of claim 36, wherein the image capture device automatically focuses on the object in accordance with the position of the audio source relative to the image capture device.
38. The system of claim 31, further comprising a lighting source external to the display device, the lighting source configured to control at least one lighting condition in the user environment in response to the consumption of bandwidth as measured by the computing device.
39. The system of claim 31, wherein the at least one lighting condition includes color.
40. The system of claim 31, wherein the at least one lighting condition includes brightness.
41. A method for managing bandwidth consumption through control of lighting conditions in a user environment, the method comprising: capturing image data in the user environment; transmitting the captured image data over a communications network; measuring consumption of bandwidth related to transmission of the captured image data over the communications network; generating an inner frame area and an outer frame area on a display device in the user environment; and controlling at least one lighting condition in the user environment, wherein the inner frame area and the outer frame area control the at least one lighting condition in the user environment in response to the consumption of bandwidth as measured by the computing device.
42. The method of claim 41, wherein controlling the at least one lighting condition includes decreasing the brightness of the user environment.
43. The method of claim 41, wherein controlling the at least one lighting condition includes adjusting the color of the user environment.
44. The method of claim 42, wherein the brightness of the user environment is controlled by the intensity of the outer frame area.
45. The method of claim 42, wherein the brightness of the user environment is controlled by the size of the outer frame area.
46. The method of claim 42, wherein the brightness of the user environment is controlled through a combination of intensity and size of the outer frame area.
47. The method of claim 46, wherein the intensity and size of the outer frame area are dynamically controlled as the at least one lighting condition in the user environment changes.
48. A computer-readable storage medium having embodied thereon a program executable by a machine, the program comprising instructions for controlling lighting conditions in a user environment, the method comprising: capturing image data in the user environment; identifying at least one lighting condition in the user environment; generating an inner frame area and an outer frame area on a display device; and controlling at least one lighting condition in the user environment through the outer frame area.
9. A video conference system, comprising: a first image capture device configured to capture first user environment image data in a first user environment for transmission over a communications network; a second image capture device configured to capture second user environment image data in a second user environment for transmission over the communications network; a conference management server coupled to the network, the conference management server configured to receive the first user environment image data and the second user environment image data, the conference management server further configured to measure consumption of bandwidth related to transmission of the first user environment image data and the second user environment image data over the communications network; a first user client computing device associated with the first image capture device, the first user client computing device configured to receive lighting control instructions from the conference management server in response to the consumption of bandwidth as measured by the conference management server; and a second user client computing device associated with the second image capture device, the second user client computing device configured to receive lighting control instructions from the conference management server in response to the consumption of bandwidth as measured by the conference management server, wherein the lighting control instructions received from the conference management server include generating an outer frame area and an inner frame area on a display device at both the first user client computing device and second user client computing device, the outer frame area and inner frame area collectively controlling at least one lighting condition in the respective user environment.
50. The video conference system of claim 49, wherein the conference management server is further configured to exchange data ancillary to the video conference with the first user client computing device and the second user client computing device.
51. The video conference system of claim 49, wherein the ancillary data includes application data.
52. The video conference system of claim 49, wherein the ancillary data includes text data.
53. The video conference server of claim 49, wherein the conference server is further configured to record image capture data exchanged between the first client computing device and the second client computing device.
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US79811206P | 2006-05-04 | 2006-05-04 | |
US60/798,112 | 2006-05-04 | ||
US11/624,886 US7965859B2 (en) | 2006-05-04 | 2007-01-19 | Lighting control of a user environment via a display device |
US11/624,886 | 2007-01-19 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2007130641A2 true WO2007130641A2 (en) | 2007-11-15 |
WO2007130641A3 WO2007130641A3 (en) | 2008-04-03 |
Family
ID=38668365
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2007/010944 WO2007130641A2 (en) | 2006-05-04 | 2007-05-04 | Lighting control of a user environment via a display device |
Country Status (2)
Country | Link |
---|---|
US (2) | US7965859B2 (en) |
WO (1) | WO2007130641A2 (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2495966A1 (en) * | 2009-10-28 | 2012-09-05 | Huawei Device Co., Ltd. | Method and apparatus for controlling conference place environment in video communication |
US9108108B2 (en) | 2007-09-05 | 2015-08-18 | Sony Computer Entertainment America Llc | Real-time, contextual display of ranked, user-generated game play advice |
US9126116B2 (en) | 2007-09-05 | 2015-09-08 | Sony Computer Entertainment America Llc | Ranking of user-generated game play advice |
US20170285594A1 (en) * | 2016-03-30 | 2017-10-05 | Lenovo (Singapore) Pte. Ltd. | Systems and methods for control of output from light output apparatus |
US9833707B2 (en) | 2012-10-29 | 2017-12-05 | Sony Interactive Entertainment Inc. | Ambient light control and calibration via a console |
US10128914B1 (en) | 2017-09-06 | 2018-11-13 | Sony Interactive Entertainment LLC | Smart tags with multiple interactions |
US10561942B2 (en) | 2017-05-15 | 2020-02-18 | Sony Interactive Entertainment America Llc | Metronome for competitive gaming headset |
Families Citing this family (49)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9061207B2 (en) * | 2002-12-10 | 2015-06-23 | Sony Computer Entertainment America Llc | Temporary decoder apparatus and method |
US8133115B2 (en) | 2003-10-22 | 2012-03-13 | Sony Computer Entertainment America Llc | System and method for recording and displaying a graphical path in a video game |
US20060071933A1 (en) | 2004-10-06 | 2006-04-06 | Sony Computer Entertainment Inc. | Application binary interface for multi-pass shaders |
US7636126B2 (en) | 2005-06-22 | 2009-12-22 | Sony Computer Entertainment Inc. | Delay matching in audio/video systems |
JPWO2007052395A1 (en) * | 2005-10-31 | 2009-04-30 | シャープ株式会社 | Viewing environment control device, viewing environment control system, viewing environment control method, data transmission device, and data transmission method |
JP4618176B2 (en) * | 2006-03-22 | 2011-01-26 | 船井電機株式会社 | Monitoring device |
US7965859B2 (en) | 2006-05-04 | 2011-06-21 | Sony Computer Entertainment Inc. | Lighting control of a user environment via a display device |
US7880746B2 (en) | 2006-05-04 | 2011-02-01 | Sony Computer Entertainment Inc. | Bandwidth management through lighting control of a user environment via a display device |
KR101411324B1 (en) * | 2007-08-14 | 2014-06-25 | 삼성전자주식회사 | Method of displaying images and Display apparatus applying the same |
US8922672B2 (en) * | 2008-01-03 | 2014-12-30 | Apple Inc. | Illumination systems and methods for imagers |
CN101848356A (en) * | 2009-03-24 | 2010-09-29 | 鸿富锦精密工业(深圳)有限公司 | Network voice and video communication system and method thereof |
US8384754B2 (en) * | 2009-06-17 | 2013-02-26 | Verizon Patent And Licensing Inc. | Method and system of providing lighting for videoconferencing |
US10786736B2 (en) | 2010-05-11 | 2020-09-29 | Sony Interactive Entertainment LLC | Placement of user information in a game space |
US8670018B2 (en) * | 2010-05-27 | 2014-03-11 | Microsoft Corporation | Detecting reactions and providing feedback to an interaction |
US8963987B2 (en) | 2010-05-27 | 2015-02-24 | Microsoft Corporation | Non-linguistic signal detection and feedback |
WO2012061549A2 (en) * | 2010-11-03 | 2012-05-10 | 3Dmedia Corporation | Methods, systems, and computer program products for creating three-dimensional video sequences |
US9342817B2 (en) | 2011-07-07 | 2016-05-17 | Sony Interactive Entertainment LLC | Auto-creating groups for sharing photos |
US9513724B2 (en) * | 2011-08-30 | 2016-12-06 | Blackberry Limited | Device and method for adjusting object illumination |
US9472163B2 (en) * | 2012-02-17 | 2016-10-18 | Monotype Imaging Inc. | Adjusting content rendering for environmental conditions |
CN103327283A (en) * | 2012-03-20 | 2013-09-25 | 捷达世软件(深圳)有限公司 | Interphone, interphone system and method for controlling brightness of interphone display panel |
US9007425B1 (en) * | 2012-08-31 | 2015-04-14 | Securus Technologies, Inc. | Software-controlled lighting for video visitation devices |
KR20140052640A (en) * | 2012-10-25 | 2014-05-07 | 삼성전자주식회사 | Method for displaying a cursor on a display and system performing the same |
CN104823432A (en) * | 2012-12-27 | 2015-08-05 | 哈曼国际工业有限公司 | Obtaining on-line service |
US9987184B2 (en) * | 2013-02-05 | 2018-06-05 | Valentin Borovinov | Systems, methods, and media for providing video of a burial memorial |
US9531985B2 (en) * | 2013-03-15 | 2016-12-27 | Samsung Electronics Co., Ltd. | Measuring user engagement of content |
JP5751270B2 (en) * | 2013-03-21 | 2015-07-22 | カシオ計算機株式会社 | Imaging apparatus, imaging control method, and program |
CN104166835A (en) * | 2013-05-17 | 2014-11-26 | 诺基亚公司 | Method and device for identifying living user |
US9525811B2 (en) | 2013-07-01 | 2016-12-20 | Qualcomm Incorporated | Display device configured as an illumination source |
KR102153436B1 (en) * | 2014-01-15 | 2020-09-08 | 엘지전자 주식회사 | Mobile terminal and method for controlling the same |
US9716861B1 (en) | 2014-03-07 | 2017-07-25 | Steelcase Inc. | Method and system for facilitating collaboration sessions |
US10664772B1 (en) | 2014-03-07 | 2020-05-26 | Steelcase Inc. | Method and system for facilitating collaboration sessions |
US9380682B2 (en) | 2014-06-05 | 2016-06-28 | Steelcase Inc. | Environment optimization for space based on presence and activities |
US9955318B1 (en) | 2014-06-05 | 2018-04-24 | Steelcase Inc. | Space guidance and management system and method |
US9766079B1 (en) | 2014-10-03 | 2017-09-19 | Steelcase Inc. | Method and system for locating resources and communicating within an enterprise |
US10433646B1 (en) | 2014-06-06 | 2019-10-08 | Steelcaase Inc. | Microclimate control systems and methods |
US11744376B2 (en) | 2014-06-06 | 2023-09-05 | Steelcase Inc. | Microclimate control systems and methods |
US9852388B1 (en) | 2014-10-03 | 2017-12-26 | Steelcase, Inc. | Method and system for locating resources and communicating within an enterprise |
CN107206281A (en) * | 2014-11-21 | 2017-09-26 | 索尼互动娱乐股份有限公司 | Program and message processing device |
US10733371B1 (en) | 2015-06-02 | 2020-08-04 | Steelcase Inc. | Template based content preparation system for use with a plurality of space types |
TWI610292B (en) * | 2015-11-19 | 2018-01-01 | 瑞鼎科技股份有限公司 | Driving circuit and operating method thereof |
US9921726B1 (en) | 2016-06-03 | 2018-03-20 | Steelcase Inc. | Smart workstation method and system |
US10264213B1 (en) | 2016-12-15 | 2019-04-16 | Steelcase Inc. | Content amplification system and method |
JP6767319B2 (en) * | 2017-07-31 | 2020-10-14 | 株式会社ソニー・インタラクティブエンタテインメント | Information processing device and file copy method |
US11254152B2 (en) * | 2017-09-15 | 2022-02-22 | Kamran Deljou | Printed frame image on artwork |
US10469723B2 (en) | 2017-10-17 | 2019-11-05 | Motorola Mobility Llc | Photography lighting control |
US10574876B2 (en) * | 2017-10-17 | 2020-02-25 | Motorola Mobility Llc | Photography environment control |
US11302289B2 (en) * | 2020-04-24 | 2022-04-12 | Rockwell Collins, Inc. | Display element, system, and method |
US11722780B2 (en) | 2021-10-19 | 2023-08-08 | Cisco Technology, Inc. | Intelligent cloud-assisted video lighting adjustments for cloud-based virtual meetings |
US11895438B2 (en) * | 2022-01-20 | 2024-02-06 | Zoom Video Communications, Inc. | Virtual background adjustment based on conference participant lighting levels |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2001082626A1 (en) * | 2000-04-13 | 2001-11-01 | Koninklijke Philips Electronics N.V. | Method and apparatus for tracking moving objects using combined video and audio information in video conferencing and other applications |
GB2411065A (en) * | 2004-02-11 | 2005-08-17 | Vodafone Plc | Mobile telephone terminal with camera and using display to illuminate scene. |
EP1630754A1 (en) * | 2004-08-30 | 2006-03-01 | Aruze Corp. | Gaming machine |
WO2006033360A1 (en) * | 2004-09-21 | 2006-03-30 | Nikon Corporation | Mobile information device |
EP1650706A2 (en) * | 2004-10-25 | 2006-04-26 | Lg Electronics Inc. | System and method for enhancing image quality of mobile communication terminal |
Family Cites Families (371)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB958588A (en) | 1960-12-08 | 1964-05-21 | Gen Electric Co Ltd | Improvements in or relating to colour television receivers |
US3147341A (en) | 1961-04-24 | 1964-09-01 | Gen Electric | Automatic brightness-contrast control using photoresistive element to control brightness and agc voltages in response to ambinent light |
US3943277A (en) | 1969-02-20 | 1976-03-09 | The United States Of America As Represented By The Secretary Of The Navy | Digital memory area correlation tracker |
US3717345A (en) | 1971-03-15 | 1973-02-20 | Prayfel Inc | Computerized race game |
NL7509871A (en) | 1975-08-20 | 1977-02-22 | Philips Nv | COLOR TV CHROMA KEY SIGNAL GENERATOR. |
US4090216A (en) | 1976-05-26 | 1978-05-16 | Gte Sylvania Incorporated | Ambient light contrast and color control circuit |
US4068847A (en) | 1976-06-23 | 1978-01-17 | The Magnavox Company | Chroma and luminance signal generator for video games |
US4116444A (en) | 1976-07-16 | 1978-09-26 | Atari, Inc. | Method for generating a plurality of moving objects on a video display screen |
US4133004A (en) | 1977-11-02 | 1979-01-02 | Hughes Aircraft Company | Video correlation tracker |
US4448200A (en) | 1978-03-27 | 1984-05-15 | University Of Southern California | System and method for dynamic background subtraction |
JPS5513582A (en) | 1978-07-13 | 1980-01-30 | Sanyo Electric Co Ltd | Color television receiver |
US4241341A (en) | 1979-03-05 | 1980-12-23 | Thorson Mark R | Apparatus for scan conversion |
US4321635A (en) | 1979-04-20 | 1982-03-23 | Teac Corporation | Apparatus for selective retrieval of information streams or items |
US4355334A (en) | 1981-05-29 | 1982-10-19 | Zenith Radio Corporation | Dimmer and dimmer override control for a display device |
JPS5846783A (en) | 1981-09-12 | 1983-03-18 | Sony Corp | Chromakey device |
JPS58195957A (en) | 1982-05-11 | 1983-11-15 | Casio Comput Co Ltd | Program starting system by voice |
US4514727A (en) | 1982-06-28 | 1985-04-30 | Trw Inc. | Automatic brightness control apparatus |
US4757525A (en) | 1982-09-29 | 1988-07-12 | Vmx, Inc. | Electronic audio communications system with voice command features |
JPS59202779A (en) | 1983-05-02 | 1984-11-16 | Matsushita Electric Ind Co Ltd | Dimmer device for self-emission type picture display |
FI68131C (en) | 1983-06-30 | 1985-07-10 | Valtion Teknillinen | REFERENCE FOR A WINDOW MACHINE WITH A GLASS LED WITH A LASER INDICATOR |
IL69327A (en) | 1983-07-26 | 1986-11-30 | Elscint Ltd | Automatic misregistration correction |
US4675562A (en) | 1983-08-01 | 1987-06-23 | Fairchild Semiconductor Corporation | Method and apparatus for dynamically controlling the timing of signals in automatic test systems |
IL72685A (en) | 1983-08-30 | 1988-08-31 | Gen Electric | Advanced video object generator |
US4646075A (en) | 1983-11-03 | 1987-02-24 | Robert Bosch Corporation | System and method for a data processing pipeline |
US4649504A (en) | 1984-05-22 | 1987-03-10 | Cae Electronics, Ltd. | Optical position and orientation measurement techniques |
US5555532A (en) | 1984-05-23 | 1996-09-10 | The United States Of America As Represented By The Secretary Of The Navy | Method and apparatus for target imaging with sidelooking sonar |
US4658247A (en) | 1984-07-30 | 1987-04-14 | Cornell Research Foundation, Inc. | Pipelined, line buffered real-time color graphics display system |
JPH0746391B2 (en) | 1984-09-14 | 1995-05-17 | 株式会社日立製作所 | Graphic seeding device |
US4672564A (en) | 1984-11-15 | 1987-06-09 | Honeywell Inc. | Method and apparatus for determining location and orientation of objects |
US4683466A (en) | 1984-12-14 | 1987-07-28 | Honeywell Information Systems Inc. | Multiple color generation on a display |
US4737921A (en) | 1985-06-03 | 1988-04-12 | Dynamic Digital Displays, Inc. | Three dimensional medical image display system |
WO1987000320A1 (en) | 1985-07-05 | 1987-01-15 | Dai Nippon Insatsu Kabushiki Kaisha | Method and apparatus for designing three-dimensional container |
JPH0814854B2 (en) | 1985-10-11 | 1996-02-14 | 株式会社日立製作所 | 3D graphic display device |
IL77610A (en) | 1986-01-15 | 1994-01-25 | Technion Res & Dev Foundation | Single camera three-dimensional head position sensing system |
US4843568A (en) | 1986-04-11 | 1989-06-27 | Krueger Myron W | Real time perception of and response to the actions of an unencumbered participant/user |
JP2603947B2 (en) | 1986-09-26 | 1997-04-23 | オリンパス光学工業株式会社 | Apparatus for detecting corresponding areas between primary color images |
US4807158A (en) | 1986-09-30 | 1989-02-21 | Daleco/Ivex Partners, Ltd. | Method and apparatus for sampling images to simulate movement within a multidimensional space |
US4905168A (en) | 1986-10-15 | 1990-02-27 | Atari Games Corporation | Object processing for video system using slips and linked list |
US4905147A (en) | 1986-10-15 | 1990-02-27 | Logg George E | Collision detection system for video system |
JPS63177193A (en) | 1987-01-19 | 1988-07-21 | 株式会社日立製作所 | Display device |
US4864515A (en) | 1987-03-30 | 1989-09-05 | Honeywell Inc. | Electronic sensing screen for measuring projectile parameters |
FR2613572B1 (en) | 1987-04-03 | 1993-01-22 | Thomson Csf | LIGHT DATA VISUALIZATION SYSTEM WITH IMPROVED READABILITY |
US5014327A (en) | 1987-06-15 | 1991-05-07 | Digital Equipment Corporation | Parallel associative memory having improved selection and decision mechanisms for recognizing and sorting relevant patterns |
US4980823A (en) | 1987-06-22 | 1990-12-25 | International Business Machines Corporation | Sequential prefetching with deconfirmation |
US4860197A (en) | 1987-07-31 | 1989-08-22 | Prime Computer, Inc. | Branch cache system with instruction boundary determination independent of parcel boundary |
US5162781A (en) | 1987-10-02 | 1992-11-10 | Automated Decisions, Inc. | Orientational mouse computer input system |
US5363120A (en) | 1987-10-14 | 1994-11-08 | Wang Laboratories, Inc. | Computer input device using orientation sensor |
US4866637A (en) | 1987-10-30 | 1989-09-12 | International Business Machines Corporation | Pipelined lighting model processing system for a graphics workstation's shading function |
US4901064A (en) | 1987-11-04 | 1990-02-13 | Schlumberger Technologies, Inc. | Normal vector shading for 3-D graphics display system |
US4942538A (en) | 1988-01-05 | 1990-07-17 | Spar Aerospace Limited | Telerobotic tracker |
US5369737A (en) | 1988-03-21 | 1994-11-29 | Digital Equipment Corporation | Normalization of vectors associated with a display pixels of computer generated images |
GB8808608D0 (en) | 1988-04-12 | 1988-05-11 | Boc Group Plc | Dry pump with booster |
US5448687A (en) | 1988-09-13 | 1995-09-05 | Computer Design, Inc. | Computer-assisted design system for flattening a three-dimensional surface and for wrapping a flat shape to a three-dimensional surface |
US4933864A (en) | 1988-10-04 | 1990-06-12 | Transitions Research Corporation | Mobile robot navigation employing ceiling light fixtures |
US5045843B1 (en) | 1988-12-06 | 1996-07-16 | Selectech Ltd | Optical pointing device |
US5222203A (en) | 1989-01-20 | 1993-06-22 | Daikin Industries, Ltd. | Method and apparatus for displaying translucent surface |
US5034986A (en) | 1989-03-01 | 1991-07-23 | Siemens Aktiengesellschaft | Method for detecting and tracking moving objects in a digital image sequence having a stationary background |
US4969036A (en) | 1989-03-31 | 1990-11-06 | Bir Bhanu | System for computing the self-motion of moving images devices |
DE69033269T2 (en) | 1989-06-20 | 2000-01-05 | Fujitsu Ltd | Method and device for measuring the position and position of an object |
US5367615A (en) | 1989-07-10 | 1994-11-22 | General Electric Company | Spatial augmentation of vertices and continuous level of detail transition for smoothly varying terrain polygon density |
FR2652972B1 (en) | 1989-10-06 | 1996-11-29 | Thomson Video Equip | METHOD AND DEVICE FOR INTEGRATING SELF-ADAPTIVE COLOR VIDEO IMAGES. |
GB9001468D0 (en) | 1990-01-23 | 1990-03-21 | Sarnoff David Res Center | Computing multiple motions within an image region |
US5668646A (en) | 1990-02-06 | 1997-09-16 | Canon Kabushiki Kaisha | Apparatus and method for decoding differently encoded multi-level and binary image data, the later corresponding to a color in the original image |
EP0441586B1 (en) | 1990-02-06 | 1996-04-24 | Canon Kabushiki Kaisha | Image processing apparatus |
US5064291A (en) | 1990-04-03 | 1991-11-12 | Hughes Aircraft Company | Method and apparatus for inspection of solder joints utilizing shape determination from shading |
US5128671A (en) | 1990-04-12 | 1992-07-07 | Ltv Aerospace And Defense Company | Control device having multiple degrees of freedom |
DE69130549T2 (en) | 1990-06-11 | 1999-05-27 | Hitachi Ltd | Device for generating an object movement path |
US5265888A (en) | 1990-06-22 | 1993-11-30 | Nintendo Co., Ltd. | Game apparatus and memory cartridge used therefor |
US5253339A (en) | 1990-07-26 | 1993-10-12 | Sun Microsystems, Inc. | Method and apparatus for adaptive Phong shading |
US5354202A (en) | 1990-08-01 | 1994-10-11 | Atari Games Corporation | System and method for driver training with multiple driver competition |
US5269687A (en) | 1990-08-01 | 1993-12-14 | Atari Games Corporation | System and method for recursive driver training |
US5208763A (en) | 1990-09-14 | 1993-05-04 | New York University | Method and apparatus for determining position and orientation of mechanical objects |
US5274560A (en) | 1990-12-03 | 1993-12-28 | Audio Navigation Systems, Inc. | Sensor free vehicle navigation system utilizing a voice input/output interface for routing a driver from his source point to his destination point |
US5268996A (en) | 1990-12-20 | 1993-12-07 | General Electric Company | Computer image generation method for determination of total pixel illumination due to plural light sources |
US5128794A (en) | 1990-12-31 | 1992-07-07 | Honeywell Inc. | Scanning laser helmet mounted sight |
US5534917A (en) | 1991-05-09 | 1996-07-09 | Very Vivid, Inc. | Video image based control system |
US5548667A (en) | 1991-05-24 | 1996-08-20 | Sony Corporation | Image processing system and method thereof in which three dimensional shape is reproduced from two dimensional image data |
US5227985A (en) | 1991-08-19 | 1993-07-13 | University Of Maryland | Computer vision system for position monitoring in three dimensions using non-coplanar light sources attached to a monitored object |
US5305389A (en) | 1991-08-30 | 1994-04-19 | Digital Equipment Corporation | Predictive cache system |
US5212888A (en) | 1991-09-16 | 1993-05-25 | Calcomp Inc. | Dual function sensor for a pen plotter |
US5537638A (en) | 1991-10-25 | 1996-07-16 | Hitachi, Ltd. | Method and system for image mapping |
US5335557A (en) | 1991-11-26 | 1994-08-09 | Taizo Yasutake | Touch sensitive input control device |
US5631697A (en) | 1991-11-27 | 1997-05-20 | Hitachi, Ltd. | Video camera capable of automatic target tracking |
US5734384A (en) | 1991-11-29 | 1998-03-31 | Picker International, Inc. | Cross-referenced sectioning and reprojection of diagnostic image volumes |
US5230623A (en) | 1991-12-10 | 1993-07-27 | Radionics, Inc. | Operating pointer with interactive computergraphics |
US5680487A (en) | 1991-12-23 | 1997-10-21 | Texas Instruments Incorporated | System and method for determining optical flow |
US5590248A (en) | 1992-01-02 | 1996-12-31 | General Electric Company | Method for reducing the complexity of a polygonal mesh |
US5953485A (en) | 1992-02-07 | 1999-09-14 | Abecassis; Max | Method and system for maintaining audio during video control |
DE69301308T2 (en) | 1992-02-18 | 1996-05-23 | Evans & Sutherland Computer Co | IMAGE TEXTURING SYSTEM WITH THEME CELLS. |
US5577179A (en) | 1992-02-25 | 1996-11-19 | Imageware Software, Inc. | Image editing system |
US5307137A (en) | 1992-03-16 | 1994-04-26 | Mark F. Jones | Terrain imaging apparatus and method |
JP3107452B2 (en) | 1992-04-28 | 2000-11-06 | 株式会社日立製作所 | Texture mapping method and apparatus |
US5450504A (en) | 1992-05-19 | 1995-09-12 | Calia; James | Method for finding a most likely matching of a target facial image in a data base of facial images |
US5366376A (en) | 1992-05-22 | 1994-11-22 | Atari Games Corporation | Driver training system and method with performance data feedback |
JP3391405B2 (en) | 1992-05-29 | 2003-03-31 | 株式会社エフ・エフ・シー | Object identification method in camera image |
US5473736A (en) | 1992-06-08 | 1995-12-05 | Chroma Graphics | Method and apparatus for ordering and remapping colors in images of real two- and three-dimensional objects |
IL102289A (en) | 1992-06-24 | 1997-08-14 | R Technologies Ltd B V | Method and system for processing moving images |
US5446714A (en) | 1992-07-21 | 1995-08-29 | Pioneer Electronic Corporation | Disc changer and player that reads and stores program data of all discs prior to reproduction and method of reproducing music on the same |
DE69331031T2 (en) | 1992-07-27 | 2002-07-04 | Matsushita Electric Ind Co Ltd | Device for parallel imaging |
US5361385A (en) | 1992-08-26 | 1994-11-01 | Reuven Bakalash | Parallel computing system for volumetric modeling, data processing and visualization |
JP3244798B2 (en) | 1992-09-08 | 2002-01-07 | 株式会社東芝 | Moving image processing device |
US5982352A (en) | 1992-09-18 | 1999-11-09 | Pryor; Timothy R. | Method for providing human input to a computer |
JP2682559B2 (en) | 1992-09-30 | 1997-11-26 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Apparatus and method for displaying image of object on display device and computer graphics display system |
GB2271259A (en) | 1992-10-02 | 1994-04-06 | Canon Res Ct Europe Ltd | Processing image data |
US5469193A (en) | 1992-10-05 | 1995-11-21 | Prelude Technology Corp. | Cordless pointing apparatus |
EP0598295B1 (en) | 1992-11-17 | 1998-10-14 | Matsushita Electric Industrial Co., Ltd. | Video and audio signal multiplexing apparatus and separating apparatus |
US5387943A (en) | 1992-12-21 | 1995-02-07 | Tektronix, Inc. | Semiautomatic lip sync recovery system |
WO1994018790A1 (en) | 1993-02-05 | 1994-08-18 | Barath, Ludwig | Environment-dependent automatic luminance control for display scrreens |
US5890122A (en) | 1993-02-08 | 1999-03-30 | Microsoft Corporation | Voice-controlled computer simulateously displaying application menu and list of available commands |
US5660547A (en) | 1993-02-17 | 1997-08-26 | Atari Games Corporation | Scenario development system for vehicle simulators |
JP3679426B2 (en) | 1993-03-15 | 2005-08-03 | マサチューセッツ・インスティチュート・オブ・テクノロジー | A system that encodes image data into multiple layers, each representing a coherent region of motion, and motion parameters associated with the layers. |
DE69424716T2 (en) | 1993-04-01 | 2001-02-08 | Sun Microsystems Inc | Method and device for adaptive control of texture mapping |
GB9308952D0 (en) | 1993-04-30 | 1993-06-16 | Philips Electronics Uk Ltd | Tracking objects in video sequences |
US5297061A (en) | 1993-05-19 | 1994-03-22 | University Of Maryland | Three dimensional pointing device monitored by computer vision |
DE69433074D1 (en) | 1993-07-02 | 2003-10-02 | Siemens Corp Res Inc | Background recovery in monocular image processing |
JPH0757117A (en) | 1993-07-09 | 1995-03-03 | Silicon Graphics Inc | Forming method of index to texture map and computer control display system |
US5550960A (en) | 1993-08-02 | 1996-08-27 | Sun Microsystems, Inc. | Method and apparatus for performing dynamic texture mapping for complex surfaces |
US5598514A (en) | 1993-08-09 | 1997-01-28 | C-Cube Microsystems | Structure and method for a multistandard video encoder/decoder |
JP2916076B2 (en) | 1993-08-26 | 1999-07-05 | シャープ株式会社 | Image display device |
DE69419291T2 (en) | 1993-09-03 | 1999-12-30 | Canon Kk | Form measuring apparatus |
JP2552427B2 (en) | 1993-12-28 | 1996-11-13 | コナミ株式会社 | Tv play system |
FR2714502A1 (en) | 1993-12-29 | 1995-06-30 | Philips Laboratoire Electroniq | An image processing method and apparatus for constructing from a source image a target image with perspective change. |
US5559950A (en) | 1994-02-02 | 1996-09-24 | Video Lottery Technologies, Inc. | Graphics processor enhancement unit |
US5699497A (en) | 1994-02-17 | 1997-12-16 | Evans & Sutherland Computer Corporation | Rendering global macro texture, for producing a dynamic image, as on computer generated terrain, seen from a moving viewpoint |
US5611000A (en) | 1994-02-22 | 1997-03-11 | Digital Equipment Corporation | Spline-based image registration |
EP0692717B1 (en) | 1994-07-15 | 2000-03-15 | Dade Chemistry Systems Inc. | Analysis instrument |
CA2198611A1 (en) | 1994-09-06 | 1996-03-14 | Arie E. Kaufman | Apparatus and method for real-time volume visualization |
US5526041A (en) | 1994-09-07 | 1996-06-11 | Sensormatic Electronics Corporation | Rail-based closed circuit T.V. surveillance system with automatic target acquisition |
US5920842A (en) | 1994-10-12 | 1999-07-06 | Pixel Instruments | Signal synchronization |
JP2642070B2 (en) | 1994-11-07 | 1997-08-20 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Method and system for generating quadrilateral mesh |
US5649032A (en) | 1994-11-14 | 1997-07-15 | David Sarnoff Research Center, Inc. | System for automatically aligning images to form a mosaic image |
EP0715453B1 (en) | 1994-11-28 | 2014-03-26 | Canon Kabushiki Kaisha | Camera controller |
JP3578498B2 (en) | 1994-12-02 | 2004-10-20 | 株式会社ソニー・コンピュータエンタテインメント | Image information processing device |
GB2295936B (en) | 1994-12-05 | 1997-02-05 | Microsoft Corp | Progressive image transmission using discrete wavelet transforms |
GB9501832D0 (en) | 1995-01-31 | 1995-03-22 | Videologic Ltd | Texturing and shading of 3-d images |
US5818553A (en) | 1995-04-10 | 1998-10-06 | Norand Corporation | Contrast control for a backlit LCD |
IL113572A (en) | 1995-05-01 | 1999-03-12 | Metalink Ltd | Symbol decoder |
US5757360A (en) | 1995-05-03 | 1998-05-26 | Mitsubishi Electric Information Technology Center America, Inc. | Hand held computer control device |
US5672820A (en) | 1995-05-16 | 1997-09-30 | Boeing North American, Inc. | Object location identification system for providing location data of an object being pointed at by a pointing device |
US5913727A (en) | 1995-06-02 | 1999-06-22 | Ahdoot; Ned | Interactive movement and contact simulation game |
US5572261A (en) | 1995-06-07 | 1996-11-05 | Cooper; J. Carl | Automatic audio to video timing measurement device and method |
US5617407A (en) | 1995-06-21 | 1997-04-01 | Bareis; Monica M. | Optical disk having speech recognition templates for information access |
US5805745A (en) | 1995-06-26 | 1998-09-08 | Lucent Technologies Inc. | Method for locating a subject's lips in a facial image |
US6002738A (en) | 1995-07-07 | 1999-12-14 | Silicon Graphics, Inc. | System and method of performing tomographic reconstruction and volume rendering using texture mapping |
US5704024A (en) | 1995-07-20 | 1997-12-30 | Silicon Graphics, Inc. | Method and an apparatus for generating reflection vectors which can be unnormalized and for using these reflection vectors to index locations on an environment map |
US5864342A (en) | 1995-08-04 | 1999-01-26 | Microsoft Corporation | Method and system for rendering graphical objects to image chunks |
US5852443A (en) | 1995-08-04 | 1998-12-22 | Microsoft Corporation | Method and system for memory decomposition in a graphics rendering system |
US5977977A (en) | 1995-08-04 | 1999-11-02 | Microsoft Corporation | Method and system for multi-pass rendering |
US6016150A (en) | 1995-08-04 | 2000-01-18 | Microsoft Corporation | Sprite compositor and method for performing lighting and shading operations using a compositor to combine factored image layers |
US5870097A (en) | 1995-08-04 | 1999-02-09 | Microsoft Corporation | Method and system for improving shadowing in a graphics rendering system |
US5808617A (en) | 1995-08-04 | 1998-09-15 | Microsoft Corporation | Method and system for depth complexity reduction in a graphics rendering system |
JP3203160B2 (en) | 1995-08-09 | 2001-08-27 | 三菱電機株式会社 | Volume rendering apparatus and method |
EP0852732A1 (en) | 1995-09-21 | 1998-07-15 | Omniplanar, Inc. | Method and apparatus for determining position and orientation |
US5959673A (en) | 1995-10-05 | 1999-09-28 | Microsoft Corporation | Transform coding of dense motion vector fields for frame and object based video coding applications |
US5818424A (en) | 1995-10-19 | 1998-10-06 | International Business Machines Corporation | Rod shaped device and data acquisition apparatus for determining the position and orientation of an object in space |
KR100261076B1 (en) | 1995-11-09 | 2000-07-01 | 윤종용 | Rendering method and apparatus of performing bump mapping and phong shading at the same time |
US5825308A (en) | 1996-11-26 | 1998-10-20 | Immersion Human Interface Corporation | Force feedback interface having isotonic and isometric functionality |
FR2741769B1 (en) | 1995-11-23 | 1997-12-19 | Thomson Broadcast Systems | PROCESS FOR PROCESSING THE SIGNAL CONSTITUTED BY A SUBJECT EVOLVING BEFORE A COLORED BACKGROUND AND DEVICE IMPLEMENTING THIS METHOD |
FR2741770B1 (en) | 1995-11-23 | 1998-01-02 | Thomson Broadcast Systems | METHOD FOR CALCULATING A CUTTING KEY OF A SUBJECT EVOLVING IN FRONT OF A COLORED BACKGROUND AND DEVICE IMPLEMENTING THIS METHOD |
US6028593A (en) | 1995-12-01 | 2000-02-22 | Immersion Corporation | Method and apparatus for providing simulated physical interactions within computer generated environments |
US5963209A (en) | 1996-01-11 | 1999-10-05 | Microsoft Corporation | Encoding and progressive transmission of progressive meshes |
US5574836A (en) | 1996-01-22 | 1996-11-12 | Broemmelsiek; Raymond M. | Interactive display apparatus and method with viewer position compensation |
JP3668547B2 (en) | 1996-01-29 | 2005-07-06 | ヤマハ株式会社 | Karaoke equipment |
US6049619A (en) | 1996-02-12 | 2000-04-11 | Sarnoff Corporation | Method and apparatus for detecting moving objects in two- and three-dimensional scenes |
AU2123297A (en) | 1996-02-12 | 1997-08-28 | Golf Age Technologies | Golf driving range distancing apparatus and methods |
US6009188A (en) | 1996-02-16 | 1999-12-28 | Microsoft Corporation | Method and system for digital plenoptic imaging |
US5982390A (en) | 1996-03-25 | 1999-11-09 | Stan Stoneking | Controlling personality manifestations by objects in a computer-assisted animation environment |
US5764803A (en) | 1996-04-03 | 1998-06-09 | Lucent Technologies Inc. | Motion-adaptive modelling of scene content for very low bit rate model-assisted coding of video sequences |
US5889505A (en) | 1996-04-04 | 1999-03-30 | Yale University | Vision-based six-degree-of-freedom computer input device |
US6348921B1 (en) | 1996-04-12 | 2002-02-19 | Ze Hong Zhao | System and method for displaying different portions of an object in different levels of detail |
AU2663597A (en) | 1996-04-12 | 1997-11-07 | Multigen, Inc. | Methods and apparatus for rendering three-dimensional images |
US5894308A (en) | 1996-04-30 | 1999-04-13 | Silicon Graphics, Inc. | Interactively reducing polygon count in three-dimensional graphic objects |
GB9609197D0 (en) | 1996-05-02 | 1996-07-03 | Philips Electronics Nv | Process control with evaluation of stored referential expressions |
US5805170A (en) | 1996-05-07 | 1998-09-08 | Microsoft Corporation | Systems and methods for wrapping a closed polygon around an object |
US5769718A (en) | 1996-05-15 | 1998-06-23 | Rieder; William R. | Video game apparatus and medium readable by a computer stored with video game program |
US6034693A (en) | 1996-05-28 | 2000-03-07 | Namco Ltd. | Image synthesizing apparatus, image synthesizing method and information storage medium |
JP3664336B2 (en) | 1996-06-25 | 2005-06-22 | 株式会社日立メディコ | Method and apparatus for setting viewpoint position and gaze direction in 3D image construction method |
EP0850673B1 (en) | 1996-07-11 | 2003-10-01 | Sega Enterprises, Ltd. | Game input device and game input method with voice recognition |
GB9616184D0 (en) | 1996-08-01 | 1996-09-11 | Philips Electronics Nv | Virtual environment navigation |
US5933150A (en) | 1996-08-06 | 1999-08-03 | Interval Research Corporation | System for image manipulation and animation using embedded constraint graphics |
US5781194A (en) | 1996-08-29 | 1998-07-14 | Animatek International, Inc. | Real-time projection of voxel-based object |
JP3358169B2 (en) | 1996-08-30 | 2002-12-16 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Mirror surface rendering method and apparatus |
JP3387750B2 (en) | 1996-09-02 | 2003-03-17 | 株式会社リコー | Shading processing equipment |
US5786801A (en) | 1996-09-06 | 1998-07-28 | Sony Corporation | Back light control apparatus and method for a flat display system |
US5854632A (en) | 1996-10-15 | 1998-12-29 | Real 3D | Apparatus and method for simulating specular reflection in a computer graphics/imaging system |
US5886702A (en) | 1996-10-16 | 1999-03-23 | Real-Time Geometry Corporation | System and method for computer modeling of 3D objects or surfaces by mesh constructions having optimal quality characteristics and dynamic resolution capabilities |
US5812136A (en) | 1996-10-30 | 1998-09-22 | Microsoft Corporation | System and method for fast rendering of a three dimensional graphical object |
US5935198A (en) | 1996-11-22 | 1999-08-10 | S3 Incorporated | Multiplier with selectable booth encoders for performing 3D graphics interpolations with two multiplies in a single pass through the multiplier |
US6157386A (en) | 1997-10-10 | 2000-12-05 | Cirrus Logic, Inc | MIP map blending in a graphics processor |
US5899810A (en) | 1997-01-24 | 1999-05-04 | Kaon Interactive Corporation | Distributed game architecture to overcome system latency |
JPH10211358A (en) | 1997-01-28 | 1998-08-11 | Sega Enterp Ltd | Game apparatus |
US6121953A (en) | 1997-02-06 | 2000-09-19 | Modern Cartoons, Ltd. | Virtual reality system for sensing facial movements |
CA2252871C (en) | 1997-02-20 | 2008-04-29 | Sony Corporation | Video signal processing device and method, image synthesizing device, and editing device |
US5870098A (en) | 1997-02-26 | 1999-02-09 | Evans & Sutherland Computer Corporation | Method for rendering shadows on a graphical display |
US5949424A (en) | 1997-02-28 | 1999-09-07 | Silicon Graphics, Inc. | Method, system, and computer program product for bump mapping in tangent space |
US5880736A (en) | 1997-02-28 | 1999-03-09 | Silicon Graphics, Inc. | Method system and computer program product for shading |
US6803964B1 (en) | 1997-03-21 | 2004-10-12 | International Business Machines Corporation | Method and apparatus for processing digital data |
US6137492A (en) | 1997-04-03 | 2000-10-24 | Microsoft Corporation | Method and system for adaptive refinement of progressive meshes |
JP4244391B2 (en) | 1997-04-04 | 2009-03-25 | ソニー株式会社 | Image conversion apparatus and image conversion method |
US6058397A (en) | 1997-04-08 | 2000-05-02 | Mitsubishi Electric Information Technology Center America, Inc. | 3D virtual environment creation management and delivery system |
US5864742A (en) | 1997-04-11 | 1999-01-26 | Eastman Kodak Company | Copy restrictive system using microdots to restrict copying of color-reversal documents |
US5917937A (en) | 1997-04-15 | 1999-06-29 | Microsoft Corporation | Method for performing stereo matching to recover depths, colors and opacities of surface elements |
US6130673A (en) | 1997-04-18 | 2000-10-10 | Silicon Graphics, Inc. | Editing a surface |
US6175367B1 (en) | 1997-04-23 | 2001-01-16 | Siligon Graphics, Inc. | Method and system for real time illumination of computer generated images |
US5912830A (en) | 1997-04-30 | 1999-06-15 | Hewlett-Packard Co. | System and method for conditionally calculating exponential values in a geometry accelerator |
US6331851B1 (en) | 1997-05-19 | 2001-12-18 | Matsushita Electric Industrial Co., Ltd. | Graphic display apparatus, synchronous reproduction method, and AV synchronous reproduction apparatus |
US5817494A (en) | 1997-05-21 | 1998-10-06 | Incyte Pharmaceuticals, Inc. | Ubiquitin conjugation proteins |
GB2326781B (en) | 1997-05-30 | 2001-10-10 | British Broadcasting Corp | Video and audio signal processing |
NL1006177C2 (en) | 1997-05-30 | 1998-12-07 | Delaval Stork V O F | Rotor shaft for a rotating machine and rotating machine provided with such a rotor shaft. |
US6222555B1 (en) | 1997-06-18 | 2001-04-24 | Christofferson Enterprises, Llc | Method for automatically smoothing object level of detail transitions for regular objects in a computer graphics display system |
US6072504A (en) | 1997-06-20 | 2000-06-06 | Lucent Technologies Inc. | Method and apparatus for tracking, storing, and synthesizing an animated version of object motion |
US6208347B1 (en) | 1997-06-23 | 2001-03-27 | Real-Time Geometry Corporation | System and method for computer modeling of 3D objects and 2D images by mesh constructions that incorporate non-spatial data such as color or texture |
US5990901A (en) | 1997-06-27 | 1999-11-23 | Microsoft Corporation | Model based image editing and correction |
US6226006B1 (en) | 1997-06-27 | 2001-05-01 | C-Light Partners, Inc. | Method and apparatus for providing shading in a graphic display system |
US6049636A (en) | 1997-06-27 | 2000-04-11 | Microsoft Corporation | Determining a rectangular box encompassing a digital picture within a digital image |
US5914724A (en) | 1997-06-30 | 1999-06-22 | Sun Microsystems, Inc | Lighting unit for a three-dimensional graphics accelerator with improved handling of incoming color values |
JP3372832B2 (en) | 1997-07-25 | 2003-02-04 | コナミ株式会社 | GAME DEVICE, GAME IMAGE PROCESSING METHOD, AND COMPUTER-READABLE RECORDING MEDIUM CONTAINING GAME IMAGE PROCESSING PROGRAM |
US6044181A (en) | 1997-08-01 | 2000-03-28 | Microsoft Corporation | Focal length estimation method and apparatus for construction of panoramic mosaic images |
US6009190A (en) | 1997-08-01 | 1999-12-28 | Microsoft Corporation | Texture map construction method and apparatus for displaying panoramic image mosaics |
US6018349A (en) | 1997-08-01 | 2000-01-25 | Microsoft Corporation | Patch-based alignment method and apparatus for construction of image mosaics |
US5987164A (en) | 1997-08-01 | 1999-11-16 | Microsoft Corporation | Block adjustment method and apparatus for construction of image mosaics |
US5986668A (en) | 1997-08-01 | 1999-11-16 | Microsoft Corporation | Deghosting method and apparatus for construction of image mosaics |
US6720949B1 (en) | 1997-08-22 | 2004-04-13 | Timothy R. Pryor | Man machine interfaces and applications |
AUPO894497A0 (en) | 1997-09-02 | 1997-09-25 | Xenotech Research Pty Ltd | Image processing method and apparatus |
US6496189B1 (en) | 1997-09-29 | 2002-12-17 | Skyline Software Systems Ltd. | Remote landscape display and pilot training |
US6072494A (en) | 1997-10-15 | 2000-06-06 | Electric Planet, Inc. | Method and apparatus for real-time gesture recognition |
US6101289A (en) | 1997-10-15 | 2000-08-08 | Electric Planet, Inc. | Method and apparatus for unencumbered capture of an object |
US6031934A (en) | 1997-10-15 | 2000-02-29 | Electric Planet, Inc. | Computer vision system for subject characterization |
US6037947A (en) | 1997-10-16 | 2000-03-14 | Sun Microsystems, Inc. | Graphics accelerator with shift count generation for handling potential fixed-point numeric overflows |
US5905894A (en) | 1997-10-29 | 1999-05-18 | Microsoft Corporation | Meta-programming methods and apparatus |
JP3890781B2 (en) | 1997-10-30 | 2007-03-07 | 株式会社セガ | Computer-readable storage medium, game device, and game image display method |
US6320580B1 (en) | 1997-11-07 | 2001-11-20 | Sega Enterprises, Ltd. | Image processing apparatus |
JP3119608B2 (en) | 1997-11-19 | 2000-12-25 | コナミ株式会社 | Competitive video game device, character movement control method in competitive video game, and recording medium storing character movement control program |
JPH11151376A (en) | 1997-11-20 | 1999-06-08 | Nintendo Co Ltd | Video game device and its storage medium |
US6162123A (en) | 1997-11-25 | 2000-12-19 | Woolston; Thomas G. | Interactive electronic sword game |
US6010403A (en) | 1997-12-05 | 2000-01-04 | Lbe Technologies, Inc. | System and method for displaying an interactive event |
ES2175326T3 (en) | 1997-12-19 | 2002-11-16 | Esab Ab | WELDING APPARATUS |
US6356288B1 (en) | 1997-12-22 | 2002-03-12 | U.S. Philips Corporation | Diversion agent uses cinematographic techniques to mask latency |
JPH11203501A (en) | 1998-01-14 | 1999-07-30 | Sega Enterp Ltd | Picture processor and picture processing method |
US6172354B1 (en) | 1998-01-28 | 2001-01-09 | Microsoft Corporation | Operator input device |
US6014144A (en) | 1998-02-03 | 2000-01-11 | Sun Microsystems, Inc. | Rapid computation of local eye vectors in a fixed point lighting unit |
US6577312B2 (en) | 1998-02-17 | 2003-06-10 | Sun Microsystems, Inc. | Graphics system configured to filter samples using a variable support filter |
US6850236B2 (en) | 1998-02-17 | 2005-02-01 | Sun Microsystems, Inc. | Dynamically adjusting a sample-to-pixel filter in response to user input and/or sensor input |
US6275187B1 (en) | 1998-03-03 | 2001-08-14 | General Electric Company | System and method for directing an adaptive antenna array |
US6181988B1 (en) | 1998-04-07 | 2001-01-30 | Raytheon Company | Guidance system having a body fixed seeker with an adjustable look angle |
US6100898A (en) | 1998-04-08 | 2000-08-08 | Webtv Networks, Inc. | System and method of selecting level of detail in texture mapping |
US6578197B1 (en) | 1998-04-08 | 2003-06-10 | Silicon Graphics, Inc. | System and method for high-speed execution of graphics application programs including shading language instructions |
US6313841B1 (en) | 1998-04-13 | 2001-11-06 | Terarecon, Inc. | Parallel volume rendering system with a resampling module for parallel and perspective projections |
JP3829014B2 (en) | 1998-06-03 | 2006-10-04 | 株式会社コナミデジタルエンタテインメント | Video game equipment |
US6573890B1 (en) | 1998-06-08 | 2003-06-03 | Microsoft Corporation | Compression of animated geometry using geometric transform coding |
US6141041A (en) | 1998-06-22 | 2000-10-31 | Lucent Technologies Inc. | Method and apparatus for determination and visualization of player field coverage in a sporting event |
JP2000090289A (en) | 1998-07-13 | 2000-03-31 | Sony Corp | Device and method for processing image and medium |
US6563499B1 (en) | 1998-07-20 | 2003-05-13 | Geometrix, Inc. | Method and apparatus for generating a 3D region from a surrounding imagery |
US6646639B1 (en) | 1998-07-22 | 2003-11-11 | Nvidia Corporation | Modified method and apparatus for improved occlusion culling in graphics systems |
GB9817834D0 (en) | 1998-08-14 | 1998-10-14 | British Telecomm | Predicting avatar movement in a distributed virtual environment |
AU5688199A (en) | 1998-08-20 | 2000-03-14 | Raycer, Inc. | System, apparatus and method for spatially sorting image data in a three-dimensional graphics pipeline |
US6771264B1 (en) | 1998-08-20 | 2004-08-03 | Apple Computer, Inc. | Method and apparatus for performing tangent space lighting and bump mapping in a deferred shading graphics processor |
US7251315B1 (en) | 1998-09-21 | 2007-07-31 | Microsoft Corporation | Speech processing for telephony API |
TW495710B (en) | 1998-10-15 | 2002-07-21 | Primax Electronics Ltd | Voice control module for control of game controller |
JP2000137828A (en) | 1998-10-30 | 2000-05-16 | Fujitsu Ltd | Shading arithmetic unit and image processor using the arithmetic unit |
GB2343598B (en) | 1998-11-06 | 2003-03-19 | Videologic Ltd | Image processing apparatus |
US6342885B1 (en) | 1998-11-12 | 2002-01-29 | Tera Recon Inc. | Method and apparatus for illuminating volume data in a rendering pipeline |
US6396490B1 (en) | 1998-12-04 | 2002-05-28 | Intel Corporation | Efficient representation of connectivity information in progressive mesh update record |
JP3748172B2 (en) | 1998-12-09 | 2006-02-22 | 富士通株式会社 | Image processing device |
JP2000181676A (en) | 1998-12-11 | 2000-06-30 | Nintendo Co Ltd | Image processor |
US6738059B1 (en) | 1998-12-18 | 2004-05-18 | Kabushiki Kaisha Sega Enterprises | Apparatus and methods for image processing using mixed display objects |
US6414960B1 (en) | 1998-12-29 | 2002-07-02 | International Business Machines Corp. | Apparatus and method of in-service audio/video synchronization testing |
US6356263B2 (en) | 1999-01-27 | 2002-03-12 | Viewpoint Corporation | Adaptive subdivision of mesh models |
JP2000218036A (en) | 1999-01-28 | 2000-08-08 | Square Co Ltd | Game device, game control method and recording medium therefor |
JP2000229172A (en) | 1999-02-10 | 2000-08-22 | Konami Co Ltd | Game system and computer readable storage medium on which game program is recorded |
JP3972230B2 (en) | 1999-02-15 | 2007-09-05 | 株式会社セガ | GAME DEVICE, GAME DEVICE CONTROL METHOD, AND RECORDING MEDIUM |
JP2000237453A (en) | 1999-02-16 | 2000-09-05 | Taito Corp | Problem solution type vehicle game device |
JP2000311251A (en) | 1999-02-26 | 2000-11-07 | Toshiba Corp | Device and method for generating animation and storage medium |
US6313842B1 (en) | 1999-03-03 | 2001-11-06 | Discreet Logic Inc. | Generating image data |
DE19917660A1 (en) | 1999-04-19 | 2000-11-02 | Deutsch Zentr Luft & Raumfahrt | Method and input device for controlling the position of an object to be graphically represented in a virtual reality |
US6226007B1 (en) | 1999-05-21 | 2001-05-01 | Sun Microsystems, Inc. | Method and apparatus for modeling specular reflection |
US6917692B1 (en) | 1999-05-25 | 2005-07-12 | Thomson Licensing S.A. | Kalman tracking of color objects |
JP3431535B2 (en) | 1999-05-26 | 2003-07-28 | 株式会社ナムコ | GAME SYSTEM AND INFORMATION STORAGE MEDIUM |
US6489955B1 (en) | 1999-06-07 | 2002-12-03 | Intel Corporation | Ray intersection reduction using directionally classified target lists |
US6717579B1 (en) | 1999-06-10 | 2004-04-06 | Dassault Systemes | Reflection line control |
US6504538B1 (en) | 1999-07-01 | 2003-01-07 | Microsoft Corporation | Method and system for generating light values for a set of vertices |
AU5935200A (en) | 1999-07-15 | 2001-02-05 | Midway Games West Inc. | System and method of vehicle competition with enhanced ghosting features |
US6421057B1 (en) | 1999-07-15 | 2002-07-16 | Terarecon, Inc. | Configurable volume rendering pipeline |
JP2001029649A (en) | 1999-07-21 | 2001-02-06 | Taito Corp | Game machine executing speech visual display by speech recognition |
US6417836B1 (en) | 1999-08-02 | 2002-07-09 | Lucent Technologies Inc. | Computer input device having six degrees of freedom for controlling movement of a three-dimensional object |
JP2001086486A (en) | 1999-09-14 | 2001-03-30 | Matsushita Electric Ind Co Ltd | Monitor camera system and display method for monitor camera |
JP2001079263A (en) | 1999-09-16 | 2001-03-27 | Sega Corp | Game device, data processing method, and information recording medium |
US6554707B1 (en) | 1999-09-24 | 2003-04-29 | Nokia Corporation | Interactive voice, wireless game system using predictive command input |
JP3270928B2 (en) | 1999-09-30 | 2002-04-02 | コナミ株式会社 | Field map generation method, video game system, and recording medium |
US6611265B1 (en) | 1999-10-18 | 2003-08-26 | S3 Graphics Co., Ltd. | Multi-stage fixed cycle pipe-lined lighting equation evaluator |
US6273814B1 (en) | 1999-10-25 | 2001-08-14 | Square Co., Ltd. | Game apparatus and method for controlling timing for executive action by game character |
US6798411B1 (en) | 1999-10-29 | 2004-09-28 | Intel Corporation | Image processing |
US6571208B1 (en) | 1999-11-29 | 2003-05-27 | Matsushita Electric Industrial Co., Ltd. | Context-dependent acoustic models for medium and large vocabulary speech recognition with eigenvoice training |
US6686924B1 (en) | 2000-02-02 | 2004-02-03 | Ati International, Srl | Method and apparatus for parallel processing of geometric aspects of video graphics data |
US6664955B1 (en) | 2000-03-15 | 2003-12-16 | Sun Microsystems, Inc. | Graphics system configured to interpolate pixel values |
US6426755B1 (en) | 2000-05-16 | 2002-07-30 | Sun Microsystems, Inc. | Graphics system using sample tags for blur |
US6594388B1 (en) | 2000-05-25 | 2003-07-15 | Eastman Kodak Company | Color image reproduction of scenes with preferential color mapping and scene-dependent tone scaling |
US6864895B1 (en) | 2000-05-30 | 2005-03-08 | Hewlett-Packard Development Company, L.P. | Pseudo-linear frame buffer mapping system and method |
US6690372B2 (en) | 2000-05-31 | 2004-02-10 | Nvidia Corporation | System, method and article of manufacture for shadow mapping |
US6717599B1 (en) | 2000-06-29 | 2004-04-06 | Microsoft Corporation | Method, system, and computer program product for implementing derivative operators with graphics hardware |
US6795068B1 (en) | 2000-07-21 | 2004-09-21 | Sony Computer Entertainment Inc. | Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program |
US6825851B1 (en) | 2000-08-23 | 2004-11-30 | Nintendo Co., Ltd. | Method and apparatus for environment-mapped bump-mapping in a graphics system |
US6316974B1 (en) | 2000-08-26 | 2001-11-13 | Rgb Systems, Inc. | Method and apparatus for vertically locking input and output signals |
US6744442B1 (en) | 2000-08-29 | 2004-06-01 | Harris Corporation | Texture mapping system used for creating three-dimensional urban models |
KR20000072753A (en) | 2000-09-25 | 2000-12-05 | 오재덕 | An intelligent toy with Wireless Local Communications. |
GB2367471B (en) | 2000-09-29 | 2002-08-14 | Pixelfusion Ltd | Graphics system |
US6853382B1 (en) | 2000-10-13 | 2005-02-08 | Nvidia Corporation | Controller for a memory system having multiple partitions |
US6765573B2 (en) | 2000-10-26 | 2004-07-20 | Square Enix Co., Ltd. | Surface shading using stored texture map based on bidirectional reflectance distribution function |
JP2002222436A (en) | 2000-11-22 | 2002-08-09 | Sony Computer Entertainment Inc | Object control method, object control processing program executed by computer, computer readable-recording medium with recorded object control processing program executed by computer, and program executing device executing object control processing program |
JP2002159749A (en) | 2000-11-28 | 2002-06-04 | Namco Ltd | Game system and program |
US6850243B1 (en) | 2000-12-07 | 2005-02-01 | Nvidia Corporation | System, method and computer program product for texture address operations based on computations involving other textures |
US6778181B1 (en) | 2000-12-07 | 2004-08-17 | Nvidia Corporation | Graphics processing system having a virtual texturing array |
US6928433B2 (en) | 2001-01-05 | 2005-08-09 | Creative Technology Ltd | Automatic hierarchical categorization of music by metadata |
US6646640B2 (en) | 2001-02-06 | 2003-11-11 | Sony Computer Entertainment Inc. | System and method for creating real-time shadows of complex transparent objects |
US6995788B2 (en) | 2001-10-10 | 2006-02-07 | Sony Computer Entertainment America Inc. | System and method for camera navigation |
US7162314B2 (en) | 2001-03-05 | 2007-01-09 | Microsoft Corporation | Scripting solution for interactive audio generation |
JP3581835B2 (en) | 2001-03-14 | 2004-10-27 | 株式会社イマジカ | Color conversion method and apparatus in chroma key processing |
US6493858B2 (en) | 2001-03-23 | 2002-12-10 | The Board Of Trustees Of The Leland Stanford Jr. University | Method and system for displaying VLSI layout data |
US6741259B2 (en) | 2001-03-30 | 2004-05-25 | Webtv Networks, Inc. | Applying multiple texture maps to objects in three-dimensional imaging processes |
US6961055B2 (en) | 2001-05-09 | 2005-11-01 | Free Radical Design Limited | Methods and apparatus for constructing virtual environments |
US7085722B2 (en) | 2001-05-14 | 2006-08-01 | Sony Computer Entertainment America Inc. | System and method for menu-driven voice control of characters in a game environment |
US6639594B2 (en) | 2001-06-03 | 2003-10-28 | Microsoft Corporation | View-dependent image synthesis |
US7162716B2 (en) | 2001-06-08 | 2007-01-09 | Nvidia Corporation | Software emulator for optimizing application-programmable vertex processing |
US7006101B1 (en) | 2001-06-08 | 2006-02-28 | Nvidia Corporation | Graphics API with branching capabilities |
US6781594B2 (en) | 2001-08-21 | 2004-08-24 | Sony Computer Entertainment America Inc. | Method for computing the intensity of specularly reflected light |
JP2003338972A (en) | 2001-09-12 | 2003-11-28 | Fuji Photo Film Co Ltd | Image processing system, imaging unit, apparatus and method for processing image and program |
EP1444605A4 (en) | 2001-10-10 | 2007-04-11 | Sony Comp Entertainment Us | Dynamically loaded game software for smooth play |
US7081893B2 (en) | 2001-10-10 | 2006-07-25 | Sony Computer Entertainment America Inc. | System and method for point pushing to render polygons in environments with changing levels of detail |
AU2002335799A1 (en) | 2001-10-10 | 2003-04-22 | Sony Computer Entertainment America Inc. | System and method for environment mapping |
JP3493189B2 (en) | 2001-10-11 | 2004-02-03 | コナミ株式会社 | Game progress control program, game progress control method, and video game apparatus |
US7443401B2 (en) | 2001-10-18 | 2008-10-28 | Microsoft Corporation | Multiple-level graphics processing with animation interval generation |
CN1292588C (en) | 2001-10-29 | 2006-12-27 | 松下电器产业株式会社 | Video-voice synchronizer |
US7034784B2 (en) | 2001-11-22 | 2006-04-25 | Sharp Kabushiki Kaisha | Optical shifter and optical display system |
JP3732168B2 (en) | 2001-12-18 | 2006-01-05 | 株式会社ソニー・コンピュータエンタテインメント | Display device, display system and display method for objects in virtual world, and method for setting land price and advertising fee in virtual world where they can be used |
US6657624B2 (en) | 2001-12-21 | 2003-12-02 | Silicon Graphics, Inc. | System, method, and computer program product for real-time shading of computer generated images |
US7212247B2 (en) | 2002-01-31 | 2007-05-01 | Thomson Licensing | Audio/video system providing variable delay |
US7159212B2 (en) | 2002-03-08 | 2007-01-02 | Electronic Arts Inc. | Systems and methods for implementing shader-driven compilation of rendering assets |
US7009605B2 (en) | 2002-03-20 | 2006-03-07 | Nvidia Corporation | System, method and computer program product for generating a shader program |
US6912010B2 (en) | 2002-04-15 | 2005-06-28 | Tektronix, Inc. | Automated lip sync error correction |
US6956871B2 (en) | 2002-04-19 | 2005-10-18 | Thomson Licensing | Apparatus and method for synchronization of audio and video streams |
US6903738B2 (en) | 2002-06-17 | 2005-06-07 | Mitsubishi Electric Research Laboratories, Inc. | Image-based 3D modeling rendering system |
US6831641B2 (en) | 2002-06-17 | 2004-12-14 | Mitsubishi Electric Research Labs, Inc. | Modeling and rendering of surface reflectance fields of 3D objects |
GB0220138D0 (en) | 2002-08-30 | 2002-10-09 | Kaydara Inc | Matte extraction using fragment processors |
US7212248B2 (en) | 2002-09-09 | 2007-05-01 | The Directv Group, Inc. | Method and apparatus for lipsync measurement and correction |
US7339589B2 (en) | 2002-10-24 | 2008-03-04 | Sony Computer Entertainment America Inc. | System and method for video choreography |
US7180529B2 (en) | 2002-12-19 | 2007-02-20 | Eastman Kodak Company | Immersive image viewing system and method |
US7072792B2 (en) | 2002-12-24 | 2006-07-04 | Daniel Freifeld | Racecourse lap counter and racecourse for radio controlled vehicles |
US7214133B2 (en) | 2003-05-09 | 2007-05-08 | Microsoft Corporation | Method and apparatus for retrieving recorded races for use in a game |
CA2527328C (en) | 2003-05-29 | 2013-04-02 | Eat.Tv, Llc | System for presentation of multimedia content |
US7428000B2 (en) | 2003-06-26 | 2008-09-23 | Microsoft Corp. | System and method for distributed meetings |
US7382369B2 (en) | 2003-10-10 | 2008-06-03 | Microsoft Corporation | Systems and methods for robust sampling for real-time relighting of objects in natural lighting environments |
US8133115B2 (en) | 2003-10-22 | 2012-03-13 | Sony Computer Entertainment America Llc | System and method for recording and displaying a graphical path in a video game |
KR100810514B1 (en) | 2003-10-28 | 2008-03-07 | 삼성전자주식회사 | Display apparatus and control method thereof |
US7570267B2 (en) | 2004-05-03 | 2009-08-04 | Microsoft Corporation | Systems and methods for providing an enhanced graphics pipeline |
US7333150B2 (en) | 2004-05-14 | 2008-02-19 | Pixel Instruments Corporation | Method, system, and program product for eliminating error contribution from production switchers with internal DVEs |
KR100703334B1 (en) * | 2004-08-20 | 2007-04-03 | 삼성전자주식회사 | Apparatus and method for displaying image in mobile terminal |
US20060071933A1 (en) | 2004-10-06 | 2006-04-06 | Sony Computer Entertainment Inc. | Application binary interface for multi-pass shaders |
US20060209210A1 (en) | 2005-03-18 | 2006-09-21 | Ati Technologies Inc. | Automatic audio and video synchronization |
US7636126B2 (en) | 2005-06-22 | 2009-12-22 | Sony Computer Entertainment Inc. | Delay matching in audio/video systems |
US7589723B2 (en) | 2005-07-25 | 2009-09-15 | Microsoft Corporation | Real-time rendering of partially translucent objects |
US9697230B2 (en) | 2005-11-09 | 2017-07-04 | Cxense Asa | Methods and apparatus for dynamic presentation of advertising, factual, and informational content using enhanced metadata in search-driven media applications |
JP2007271999A (en) | 2006-03-31 | 2007-10-18 | Renesas Technology Corp | Photomask, method for manufacturing the same, and method for manufacturing electronic device |
US7965859B2 (en) | 2006-05-04 | 2011-06-21 | Sony Computer Entertainment Inc. | Lighting control of a user environment via a display device |
US7880746B2 (en) | 2006-05-04 | 2011-02-01 | Sony Computer Entertainment Inc. | Bandwidth management through lighting control of a user environment via a display device |
JP2007334664A (en) | 2006-06-15 | 2007-12-27 | Sharp Corp | Process controller, process control method, and computer program |
US7907117B2 (en) | 2006-08-08 | 2011-03-15 | Microsoft Corporation | Virtual controller for visual displays |
US7896733B2 (en) | 2006-09-14 | 2011-03-01 | Nintendo Co., Ltd. | Method and apparatus for providing interesting and exciting video game play using a stability/energy meter |
US20080111789A1 (en) | 2006-11-09 | 2008-05-15 | Intelligence Frontier Media Laboratory Ltd | Control device with hybrid sensing system comprised of video-based pattern recognition and electronic signal transmission |
KR100849693B1 (en) | 2006-12-28 | 2008-07-31 | 엔에이치엔(주) | Method and system for generating static-shadow |
JP5285234B2 (en) | 2007-04-24 | 2013-09-11 | 任天堂株式会社 | Game system, information processing system |
US8566158B2 (en) | 2008-01-28 | 2013-10-22 | At&T Intellectual Property I, Lp | System and method for harvesting advertising data for dynamic placement into end user data streams |
-
2007
- 2007-01-19 US US11/624,886 patent/US7965859B2/en active Active
- 2007-05-04 WO PCT/US2007/010944 patent/WO2007130641A2/en active Application Filing
-
2011
- 2011-06-17 US US13/163,621 patent/US8204272B2/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2001082626A1 (en) * | 2000-04-13 | 2001-11-01 | Koninklijke Philips Electronics N.V. | Method and apparatus for tracking moving objects using combined video and audio information in video conferencing and other applications |
GB2411065A (en) * | 2004-02-11 | 2005-08-17 | Vodafone Plc | Mobile telephone terminal with camera and using display to illuminate scene. |
EP1630754A1 (en) * | 2004-08-30 | 2006-03-01 | Aruze Corp. | Gaming machine |
WO2006033360A1 (en) * | 2004-09-21 | 2006-03-30 | Nikon Corporation | Mobile information device |
EP1650706A2 (en) * | 2004-10-25 | 2006-04-26 | Lg Electronics Inc. | System and method for enhancing image quality of mobile communication terminal |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10486069B2 (en) | 2007-09-05 | 2019-11-26 | Sony Interactive Entertainment America Llc | Ranking of user-generated game play advice |
US9108108B2 (en) | 2007-09-05 | 2015-08-18 | Sony Computer Entertainment America Llc | Real-time, contextual display of ranked, user-generated game play advice |
US9126116B2 (en) | 2007-09-05 | 2015-09-08 | Sony Computer Entertainment America Llc | Ranking of user-generated game play advice |
EP2495966A4 (en) * | 2009-10-28 | 2013-06-26 | Huawei Device Co Ltd | Method and apparatus for controlling conference place environment in video communication |
US8866875B2 (en) | 2009-10-28 | 2014-10-21 | Huawei Device Co., Ltd. | Method and apparatus for controlling a conference place environment in video communication |
EP2495966A1 (en) * | 2009-10-28 | 2012-09-05 | Huawei Device Co., Ltd. | Method and apparatus for controlling conference place environment in video communication |
US9833707B2 (en) | 2012-10-29 | 2017-12-05 | Sony Interactive Entertainment Inc. | Ambient light control and calibration via a console |
US9950259B2 (en) | 2012-10-29 | 2018-04-24 | Sony Interactive Entertainment Inc. | Ambient light control and calibration via a console |
US20170285594A1 (en) * | 2016-03-30 | 2017-10-05 | Lenovo (Singapore) Pte. Ltd. | Systems and methods for control of output from light output apparatus |
US10719059B2 (en) * | 2016-03-30 | 2020-07-21 | Lenovo (Singapore) Pte. Ltd. | Systems and methods for control of output from light output apparatus |
US10561942B2 (en) | 2017-05-15 | 2020-02-18 | Sony Interactive Entertainment America Llc | Metronome for competitive gaming headset |
US10128914B1 (en) | 2017-09-06 | 2018-11-13 | Sony Interactive Entertainment LLC | Smart tags with multiple interactions |
US10541731B2 (en) | 2017-09-06 | 2020-01-21 | Sony Interactive Entertainment LLC | Smart tags with multiple interactions |
Also Published As
Publication number | Publication date |
---|---|
US20070279427A1 (en) | 2007-12-06 |
US7965859B2 (en) | 2011-06-21 |
US8204272B2 (en) | 2012-06-19 |
US20110249072A1 (en) | 2011-10-13 |
WO2007130641A3 (en) | 2008-04-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7880746B2 (en) | Bandwidth management through lighting control of a user environment via a display device | |
US7965859B2 (en) | Lighting control of a user environment via a display device | |
CN105744342B (en) | The data transmission method and device of mobile terminal | |
US7982762B2 (en) | System and method for combining local and remote images such that images of participants appear overlaid on another in substanial alignment | |
US8063992B2 (en) | Dominant color extraction for ambient light derived from video content mapped through unrendered color space | |
WO2018120946A1 (en) | Method and apparatus for determining video image abnormality, and terminal device | |
US7859595B2 (en) | Flicker-free adaptive thresholding for ambient light derived from video content mapped through unrendered color space | |
US10805680B2 (en) | Method and device for configuring image mode | |
CN106134188B (en) | Elementary video bitstream analysis | |
CN114677298A (en) | Lamp effect control method and control terminal | |
US20220319015A1 (en) | Selecting an image analysis area based on a comparison of dynamicity levels | |
CN108205795A (en) | Face image processing process and system during a kind of live streaming | |
Reznik | User-adaptive mobile video streaming using MPEG-DASH | |
CN112995565B (en) | Camera adjustment method of display device, display device and storage medium | |
CN112839252B (en) | Display device | |
CN115527474A (en) | Image display method, image display device, projection device, and storage medium | |
EP4042405A1 (en) | Perceptually improved color display in image sequences on physical displays | |
US10200662B2 (en) | Color corrected images for projectors | |
CN101521827B (en) | System and method for automatically adjusting cross-color artifact inhibition threshold | |
CN110602530A (en) | Video transmission processing method, device, server and readable storage medium | |
CN117389498A (en) | Image processing method, system, device and storage medium based on screen projection | |
CN117321626A (en) | Content optimized ambient light HDR video adaptation | |
CN117221293A (en) | Data transmission method, device, electronic equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 07776798 Country of ref document: EP Kind code of ref document: A2 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 07776798 Country of ref document: EP Kind code of ref document: A2 |