CA2157613C - Video conference system and method of providing parallax correction and a sense of presence - Google Patents
Video conference system and method of providing parallax correction and a sense of presenceInfo
- Publication number
- CA2157613C CA2157613C CA002157613A CA2157613A CA2157613C CA 2157613 C CA2157613 C CA 2157613C CA 002157613 A CA002157613 A CA 002157613A CA 2157613 A CA2157613 A CA 2157613A CA 2157613 C CA2157613 C CA 2157613C
- Authority
- CA
- Canada
- Prior art keywords
- conferee
- image
- parallax
- video
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/142—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
- H04N7/144—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
Abstract
A video conference system provides eye contact and a sense of presence to a plurality of conference participants (A, B) located in respective remotely-sited conference rooms (26, 28). Each conference room contains at least one video telephone or communications device that includes a video camera (30, 32) for generating video signals indicative of a sequence of local conferee image frames, and an image receiver (22, 24) for displaying image frames of at least one remote conferee. The image receiver, the video camera, and the eyes of the local conferee define a parallax angle (?). The video conference system further includes a frame generating system (34), responsive to the video signals, for analyzing local conferee image frames and generating a corresponding sequence of parallax-compensated frames. A signal indicative of each respective sequence of parallax-compensated frames is transmitted to a corresponding image receiver, whereby apparent eye contact is provided between each local conferee and a displayed image of a corresponding remote conferee. Where more than three conferees are involved, each input image frame is additionally analyzed for head position and the head position is reoriented by the frame generating system as necessary to provide a sense of presence.
Description
VIDEO CONFERENCE SYSTEM AND METHOD
OF PROVIDING PARALLAX CORRECTION
AND A SENSE OF PRESENCE
s Field of the Invention The present invention relates generally to video conferencing, and more particularly, to a video conference system and technique in which the appearance of eye contact is provided between each viewer and the corresponding image or images ofremote persons being viewed.
lo B~ k~round of the Invention One of the main objectives of a video conferencing system is to provide an effective vehicle for interaction between remote persons. The ability to see and interact with difference conference participants is essential to the conduct of any meeting. Visual presence is therefore one of the most important criteria a video conferencing system must 1S satisfy. It is also desirable to see who is looking at whom during the meeting.
All video conference systems utilize, at each site, at least one monitor and at least one camera for displaying the remote conferee(s) and for obtaining an image of the local conferee(s). A persistent problem which arises from current configurations of these components in a video conferencing system is the so-called "parallax effect", which 20 prevents the appearance of eye contact between the speaker and those remote viewers to whom he or she is speaking. Essentially, this effect results from the placement of the camera relative to the monitor and viewer.
The parallax effect is illustrated in FIG. 1. As seen in FIG. 1, each of conference participants A and B views a respective display 12, 14, having a window for presenting the 2s image of the other participant. Associated with each display is a video camera for capturing the image of the corresponding viewer. Each camera may be placed above, below, or to either lateral side of the associated display. In the illustrative prior art system shown in FIG. 1, cameras 16 and 18, respectively, are placed directly above displays 12 and 14. The angle ~ represents the angle between camera 16 and display 12 relative to 30 viewer A. Because participant A is looking directly at the displayed image of participant B, and therefore not at camera 16, the displayed image of A appears to be looking down.
The parallax effect has a similar impact on the displayed image of participant B.
As will be readily ascertained by those skilled in the art, the parallax effect can be minimi7Pd by placing the camera and monitor as close together as possible. In fact, 3s previous efforts to elimin~e the parallax effect have relied upon positioning the camera directly behind a specially constructed display screen. In U.S. Patent No. 4,054,908 issued to Poieier et al on October 18, 1977, for example, the image of the observer is reflected to the camera lens by an oblique reflecting plate. The display screen is viewed by the observer through a window in the reflecting plate. The presence of the window in the apertured plate, however, is somewhat distracting to the viewer. It has also been proposed to utilize an electronic crystal screen which alternates between opaque and transparent states at a frequency slot enough to permit an image of the viewer to reach the camera while in the transparent state and to present a displayed image while in the opaque, reflecting state. Disadvantageously, images displayed on such a flashing screen are characterized by unacceptably low brightness. Moreover, the screen structure andassociated driving circuitry are costly to fabricate.
So-called virtual space video conferencing systems have been proposed in an effort to provide full visual presence and convey the spatial relationship between three or more conference participants. In this type of system, the number of monitor and camera pairs at each site corresponds to the total number of remote conference participants, and the 1S relative positions of the pairs relative to each participant are fixed at each site. An illustrative system of this type is depicted in FIG. 2. Each viewing subject (A,B,C, and D) sits in a swivel chair in a corresponding room ( 1,2,3, or 4) with three TV monitors representing the other three "virtually present" participants. Thus, for example, the viewing subject (A) sits in a swivel chair in Room 1 with three TV monitors representing the three "virtually present" participants (B, C, and D). The arrangement of monitor/camera pairs in each room duplicates the positions of each participant as if they were all seated at the same square table. The signals from these cameras pass through video switching units and are directed to corresponding destinations monitors by a dynamic allocation network.
2s The virtual space system of FIG. 2 provides three or more participants with the ability to see who is looking at whom during a conference. Eye contact between the participants, however, has heretofore been obtained by resorting to the apertured or semi-reflecting screen devices discussed above. Moreover, the large number of video channels needed to provide the desired spatial distribution between the conference participants translates into a large expenditure of transmission bandwidth.
Summar~ of the Invention The system and process of the present invention overcomes the aforementioned disadvantages by using real-time image analysis and processing techniques to provide me~ningful head orientation and/or eye contact in the images presented to the remote 3s viewer. A video conferencing system constructed in accordance with the present invention includes, in each conference room, at least one video telephone that include a video camera for generating video signals indicative of a sequence of local conferee images frames, and an image receiver for displaying image frames of at least one remote conferee. The image receiver, the video camera, and the eyes of the local conferee define a parallax angle. The video conference system further includes a frame generating system, 5 responsive to the video signals, for analyzing local conferee image frames and generating a corresponding sequence of parallax compensated frames. A signal indicative of each respective sequence of parallax compensated frames is tr~nsmitted to a corresponding image receiver, whereby eye contact is provided between each local conferee and a displayed image of a corresponding remote conferee.
If more than two conferees are involved, each input image frame is analyzed for head position and reoriented, as necessary, by the frame generating means to provide a sense of presence. Illustratively, each image receiver may present two remote conferees on a split window display. Each input image frame is mapped onto a three dimensional ellipsoid head model, reoriented by a predetermined angle to obtain the proper gaze lS direction, and recapped onto a two dimensional reconstructed image frame. A signal representing the reconstructed image frames is then presented to the appr~,pliate remote image receivers. Thus, in accordance with the present invention, it is possible for each conferee to discern who is looking at whom during the conference.
Other features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims.
Brief Description of the Drawin~
2s In the drawings, wherein like reference characters denote similar elements throughout the several views:
FIG. 1 depicts the operation of a conventional single person camera (SPC) video conference system;
FIG. 2 depicts a conventional virtual space video conferencing system;
F~G. 3A is a block diagram depicting a video conferencing system constructed in accordance with an illustrative embodiment of the present invention;
FIG. 3B is an elevational view depicting the video conferencing system of FIG.
3A;
F~G. 4 is a graphical representation of a look up table for providing pixel values in accordance with an eye synthesis technique utilized by the present invention;
4 ~lS7613 FIG. S is a flowchart depicting the steps of an inventive image analysis and processing technique by which parallax correction is provided in accordance with the present invention;
FIG. 6 is an elevational view illustrating the operation of a video conferencings system providing a sense of presence between three participants in accordance with the invention; and FIG. 7 is a flowchart depicting the steps of a modified image analysis and processing technique by which parallax correction and head reorientation are performed to reproduce a sense of presence during operation of the system of FIG. 5.
0 Detailed Descfiption A conferencing system 20 constructed in accordance with an illustrative embodiment of the present invention is depicted in FIG. 3A. As soon in FM. 3A, each participant in a video teleconference views a respective display 22, 24. Each display may be divided into one or more windows for presenting the image of a corresponding number 15 of remote participants. Illustratively, there by be two conference sites with one conference participant at each site. Thus, in FIG. 3A, participants A and B are shown both a corresponding conference sites 26, 28 and as they appear on displays 22 and 24, respectively.
In the illustrative embodiment depicted in FIG. 3A, the images of conference 20 participants A and B are initially obtained by video cameras 30 and 32, respectively. In a conventional manner, the analog output signals of cameras 30 and 32 are converted to a digital form, pre-processed, and encoded for bit rate reduction to facilitate tr~ncmi~ion over or via a telecommunication network (not shown). Systems and techniques for performing the aforementioned signal processing functions are well known and are not 2s deemed to constitute a novel aspect of the present invention; accordingly, a detailed description of the same is believed unnPces.s:~ry and has been omitted.
With continued reference to FIG. 3A, it can be seen that the digitized video signals from each camera are not tr~nsmitted directly to the associated destination display.
Rather, and for a purpose which will now be explained, the outputs of cameras 30 and 32 30 are supplied to a suitable processing device 34, such for example as a high processing speed computer. Essentially, processing device 34 provides frame by frame analysis, in real time, of the raw incoming video that that it receives and modifies the image of each incoming frame so as to restore eye contact between the displayed image and the observer.
The modified frames are then tr~n~mitted, in real time, to the appropriate destination 35 display.
As was discussed above with reference to FIG. 1, the positioning of the camera directly above the display results in the formation of a parallax angle ~ between the camera, the eyes of observer A, and the displayed image of participant B. This arrangement of camera, display, and observer is repeated in FIG. 3B. Although each 5 camera is shown as being positioned above its corresponding display, it will be readily appreciated by those skilled in the art that the cameras may be positioned below or at either lateral side of the display, as well. In accordance with the present invention, parallax correction is performed by procescing device 34. For this purpose, it is first necessary to determine the parallax angle ~.
In order of increasing precision, there are several ways to determine the parallax angle ~. In the illustrative embodiment of FIGS. 3A and 3B, the images of only two conference participants, looking straight ahead, need be modified. Assuming the movements of A and B are negligible, a fixed angle ~ can be determined based on the relative positions of the camera and the display. ~lttqrn~tively, if it is assumed that the 15 image of B on display A does not move very much, then the image taken by camera A may be analyzed to determine the amount of correction needed for A's image to face straight ahead. The amount of correction required, denoted as ~, can be modeled as a random variable with a certain probability distribution, stationary or dme varying. With the assumption that A is looking at the image of B most of the time, the parallax angle ~ can 20 be equated with the expectation value of ~, i.e. E[~].
When employing either of the above-mentioned lack-of-movement assumptions, it is only n~cess:~ry to locate the eyes in each image frame and to modify the frames to correct for the parallax angle ~. Feature extraction techniques for locating facial features such as the eyes, nose and mouth in a digitized image are well known and are not2s described in detail here. Similarly, techniques for modifying localized areas of such images are known to those skilled in this art. In any event, because the eyes occupy only a small portion of the entire face or image thereof, the task of turning the eyes forward can be done by simply synthesizing the eyes. The feasibility of eye synthesis relies on the natural tolerance of the human viewer to mirror noise in the relatively small region of the eyes. In 30 this regard, it should be noted that the position of the camera relative to the display affects the complexity of the eye synthesis algorithm as well as the quality of the results obtained.
Specifically, the eye lid positions are significantly affected by the vertical component of the gaze direction, so that if the eyes had to be moved vertically, as would those in the displayed image of G, the lid would also have to be altered to avoid perceptual 3s incongruity. With the camera to the side of the display, lid position does not have to change as the eye is synthesized. Accordingly, a camera position to the side of the display is especially preferred.
In order to synthPsi7P an image of the eye, the program must determine the boundary of the surrounding lids. The lids primarily adjoin the white, but often also S adjoining the iris at the top and bottom. The entire region within this boundary is synthesized. If the subject is far away from the camera (so that the eyes in the obtained image are very small), the position of the pupil needed to provide eye contact with the observer can be established merely by adding a properly size, dark circular region at the corrected position.
If the subject is sufficiently close to the camera/monitor, completely new intensity values may be created for each pixel position within the outline of the eye, that is, between the upper and lower lids. In addition to the eye' s outline, five additional parameters are utilized to synthesiæ a new eye: the two coordinates of the pupil's center, the iris radius, the iris intensity level, and the white's intensity level. A boundary detection method 15 naturally results in a simple boundary representation containing the minimum and maximum iris X coordinates for each Y value within a range. The iris intensity is computed during the basic eye tracking algorithm, and similarly the white intensity and eye outline are computed during the eye white boundary deterrnin~tion. Illustratively, eye synthesis may be performed in accordance with a parametric mixing function approach in 20 which the iris diameter is proportional to the eye-to-eye distance. The eye-to-eye distance provides the proper scaling as the subject moves closer to and farther away from the camera.
As discussed above, a fixed angle ~ can be determined based on the relative positions of the camera and the display. Illustratively, the camera may be positioned to 25 one side of the display and at the same height. For purposes of the present invention, it may be assumed that the user is looking at the display and not somewhere else. Thus, if the user looks somewhere else, the synthe~i7ed image no longer appears to m~int~in eye contact with the viewer. With the camera and display arranged in this manner, the iris and pupil may be shifted horizontally by a certain geometry-dependent amount, with no 30 vertical motion. The magnitude of the shift is inversely proportional to the eye-to-eye distance. Illustratively, the synthesi7ing algorithm may be configured to compute the distance to the synth~.ciæd pupil center, in aspect-ratio-corrected horizontal pixels. A
look up table, based on the ratio of this distance to the synthesized iris radius, supplies a mixing coefficient between the iris level and the white level: 1 specifies the iris level and 0 35 specifies the white level, with linear mixing in between. A graphical representation of the -look-up table is shown in FM. 4. The entries above 1 produce an even darker level for the pupil. The synthesis of the white may utilize a similar process.
As will be readily appreciated by those skilled in the art, an eye image synthe~i~Pd in the above described manner will lack the reflected highlight which gives the eye an s impression of depth and contour. As such, it may be desirable to add such highlights to the image in a conventional manner. Utilizing a Motorola 88000 RISC processor toperform the aforementioned image analysis and processing operations on a single input video frame, it took 1.177 msec to locate the pupil and white boundaries, 0.176 msec to post process the boundaries, 1.670 msec to synthesize the new eyes, and 1.453 msec to 0 generate real-time graphics and text displays. Of course, the processor times are sensitive to the eye's image in the image plane. However, even with extreme close-ups, it is possible to generate a modified frame having synthesiæd eyes within one frame interval.
As a result, a sequence of parallax compensated image frame may be generated in real-time to advantageously provide the users of a video conferencing system with eye contact S without sacrifices in frame rate.
It may not be possible to utilize the aforementioned assumptions as to the movements of A and B. If compensation for the exact parallax angle ~ is required, it is necessary to detect A's eye position and B's eye position on display 24. Any suitable image analysis technique may be utilized to detect the eye positions for the purposes of the 20 present invention. By way of illustrative example only, a non-intrusive gaze tracking technique that may be utilized to obtain eye position information is described by S. Baluja and D. A. Pomerleau in an article entitled "Non-Intrusive Gaze Tracking Using Artificial Neural Networks", 6 Advances in Neural Information Processing Systems (NIPS) (1994).
It should be noted that a non-intrusive gaze technique is preferred so that each conference 2s participant may move his or her head freely.
Essentially, the goal of gaze tracking is to determine where a subject is looking from the appearance of the subject's eye. In order to account for shifts in the relative positions of the camera and the eye, the eye is first located in each image frame.
Typically, one of the eyes is located by searching for the specular reflection of a stationary 30 light source in the image of the subject's face. Although any suitable light source may be utilized for the purposes of the present invention, a non-intrusive infrared source is preferred since it provides an easily detected reflection pattern without creating a distraction or otherwise causing discomfort to the viewer. The specular reflection in the image can usually be distinguished by a small bright region surrounded by a very dark 3s region. The reflection location may be utilized to limit the search fro the eye in the next frame. A window surrounding the reflection is extracted and the image of the eye is located within this window. The center of the pupil is found and the relative position of the light's reflection to the pupil's center is calculated. The gaze direction is determined from information about the relative positions. For example, when the observer is looking at the light source, the specular reflection is at the center of the pupil image. If the observer is looking above the light, the specular reflection is near the bottom edge of the pupil image. If the observer is looking to the left of the light, the specular reflection is near the right of the pupil image, and so on.
In the illustrative system depicted in FIGS. 3A and 3B, a fixed light source 36 is positioned above camera 32 to facilitate the use of a gaze tracking technique on the image of participant A. A similar light source (not shown) is positioned above camera 30. As will be readily appreciated by those skilled in the art, a gaze detection technique such as the one discussed above provides useful information concerning the direction of the eye relative to the light source. As will not be explained, this information may be used to determine the parallax angle ~. Specifically, and as best seen in FIG. 3B, light source 36, participant A, and the displayed image of participant B define angle ~317 while the light source 36, participant A, and camera 32 define angle 2- Parallax angle 0 may thus be calculated by taking the difference between l and 2- Once the parallax angle is known, generation and tr~nsmiccion of a revised image frame is performed in the same manner as described above.
An illustrative sequence of steps for performing parallax correction with a video conferencing system such as that depicted in FIGS. 3A and 3B is shown in FIG. 5.Although the discussion will proceed in connection with the processing of images of only one participant, it will be readily apparent that the same image analysis and processing steps are simultaneously performed for all conferee participants. In any event, a digitized video frame of participant A is received by processing device 34 (step 40) and image feature extraction is performed to locate the eyes in the image (step 42). During step 44, gaze detection is performed on the image to detect the direction in which the subject is looking relative to the camera. The parallax angle is derived (step 46) and a revised image frame is generated (step 48) in which the positions of the pupils and, if appropliate, the irises correspond with the positions need for eye contact with observing participant B.
The revised image frame is then tr~ncmitted to display 22 (step 50) where it is viewed by participant B. The aforementioned steps are performed, in real time and frame by frame for each video signal input, so that eye contact is provided between the displayed images and the respective participants.
When more than two conference sites and/or participants are involved, additionalconsiderations besides the need to provide eye contact between the participants must be addressed in order for a video conferencing system to preserve the look and feel of a live meeting. Specifically, and as already discussed above, it is also important for each participant to see who is talking to whom. Thus, in accordance with the teachings of the present invention, real-time image analysis and processing techniques are utilized to s reorient the head image as well as to provide the conference participants with the appearance of eye contact.
FIG. 6 again depicts a video conferencing system 60 constructed in accordance with the present invention and similar to the system 20 of FIG. 3A. The system illustrated in FIG. 6, however, has been modified to provide a conference between three participants 10 A, B, and C at three different sites. System 60 therefore includes three respective displays 62, 64, and 66, and three corresponding cameras 68, 70, and 72. A light source, only one of which is shown and indicated generally at 76, is positioned above each camera Each of the displays are divided into two windows, with each window presenting the image of two remote participants. In a manner identical to that discussed above, the video signals from 5 each camera are digitized, processed and transmitted to a processing device such as s the processing device 74.
At the point of the video teleconference depicted in FIG. 6, A is speaking to B. In an actual conference, A would be looking at B, and both B and C would be looking at A.
Such each participant, absent the additional image processing of the invention, is facing 20 straight ahead at his or her display, however, the respective images presented to the other participants will not preserve the corresponding head orientations. To obtain the proper orientations depicted in FIG. 6, varying amounts of image analysis and processing must be performed, as will now be described.
While A is looking at the window of B on display 62, A should see the images of B
2s and C facing straight ahead (i.e. facing A). Since B and C are already facing their own cameras, only parallax error correction of their eye positions is required. Such error correcdon may be performed utilizing any of the techniques described above in connecdon with FIGS. 3A and 3B. On display 64, B should see the image of A facing straight ahead, and the image of C facing the image of A. While only parallax error correction of A's 30 image is required for display 64, the image of C additionally requires reorientation to achieve the position shown in FIG. 6. Similarly, on display 66, C should see A and B
facing each other. Thus, the images of both A and B require at least head reorientation to achieve the posidons depicted in FIG. 6.
A process for performing head reorientation and parallax correction in accordance 35 with an illustrative embodiment of the present invention is depicted in the flow chart of FIG. 7. A digidzed video frame of a conference participant such, for example, as lo 2157613 participant C is received by processing device 74 (step 80) and image feature extraction is performed to locate the eyes in the image (step 82). During step 84, gaze detection is performed on the image to detect the direction in which the subject is looking relative to the camera. When participant C is looking at participant A, for example, the specular s reflection of light source 76, in the image of participant C's eye as obtained by camera 72, will be above and to the right of the center of the pupil.
As discussed above, the parallax angle 0 can be computed from this information given the fixed position of camera 72 and display 66 (step 86). More significantly, however, the position of the specular reflection relative to the pupil center may be utilized to detect which image is being watched by the observing participant (step 88). In the present example, the location of the specular reflection to the right of the pupil center in the received image of participant C suggests that C is looking to the left, and therefore at participant A.
Once it is known which display window is being viewed by each conference participant, image processing is performed to generate respective, corrected image frames that are apprupliate for each destination display. If only parallax correction is required, as determined at step 88, a first frame is generated for transmission to the applopflate display(s) (step 90). In the illustrative example depicted in FIG. 6, a frame indicative of the parallax corrected, forward-facing image of C is presented to display 62. Because reorientation of C's head is required to convey a sense of presence to participant B, however, several additional proces~ing steps must now be performed.
The face image of each frame is texture mapped onto a three dimensional (3-D) head model (step 92). Illustratively, a relatively simple ellipsoid head model may be employed for this purpose. Of course, it will be readily appreciated by those skilled in the 2s art that other, more sophisticated 3-D models, may also be employed, depending upon the processing power and speed of execution limitations imposed by the specific application.
By way of additional example, a wire frame model as disclosed by K. Aizawa, H.
Harashima, and T. Saito in a paper entitled "Model-based Analysis Synthesis Image Coding System for a Person's Face", Signal Processing Image Communication, Vol. 1, No. 2, October 1989, pp. 139-152, may be utilized. In any event, the resulting three-dimensional, textured model is rotated in the appropriate direction by a predetermined angle a (step 94) and the texture of the ellipsoid model is projected back to the 2D
viewing plane to produce the revised image (step 96). The thus generated image is then transmitted to the appropriate destination display(s), as shown in step 98. In the 3s illustrative example depicted in FIG. 6, the image of C presented to display 64 has been rotated 90 to the right to face in the direction of the displayed image of participant A.
The eyeballs of image C can be independently modeled in a similar manner or synthesi7ed in connection with the pure parallax correction technique discussed above.
Along with video coding, parallax correction and/or head orientation can be performed at the respective conference sites as preprocessing or post-processing. In s either case, image analysis is preferably performed before coding of the original image signals so that no critical image information is lost prior to analysis. Synthesis of images corrected in accordance with the present invention, however, may be performed anywhere.
In the pre-processing case, the image is digitized, analyzed, synthesi7ed/processed, coded, and tr~n~mi~ted. Because different processed images must be sent to different 10 participants, a large amount of tr~n~mic.cion bandwidth is required. In the post-processing case, the parallax angle and head orientation information of each participant obtained by image analysis must be sent along with the coded video signals. Synthesis may then be performed at the receiver end, or in some centralized location.
While there have been shown and described and pointed out fundamental novel 5 features of the invention as applied to preferred embodiments thereof, it will be understood that various omissions and substitutions and changes in the form and details of the disclosed invention may be made by those skilled in the art without departing from the spirit of the invention. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.
OF PROVIDING PARALLAX CORRECTION
AND A SENSE OF PRESENCE
s Field of the Invention The present invention relates generally to video conferencing, and more particularly, to a video conference system and technique in which the appearance of eye contact is provided between each viewer and the corresponding image or images ofremote persons being viewed.
lo B~ k~round of the Invention One of the main objectives of a video conferencing system is to provide an effective vehicle for interaction between remote persons. The ability to see and interact with difference conference participants is essential to the conduct of any meeting. Visual presence is therefore one of the most important criteria a video conferencing system must 1S satisfy. It is also desirable to see who is looking at whom during the meeting.
All video conference systems utilize, at each site, at least one monitor and at least one camera for displaying the remote conferee(s) and for obtaining an image of the local conferee(s). A persistent problem which arises from current configurations of these components in a video conferencing system is the so-called "parallax effect", which 20 prevents the appearance of eye contact between the speaker and those remote viewers to whom he or she is speaking. Essentially, this effect results from the placement of the camera relative to the monitor and viewer.
The parallax effect is illustrated in FIG. 1. As seen in FIG. 1, each of conference participants A and B views a respective display 12, 14, having a window for presenting the 2s image of the other participant. Associated with each display is a video camera for capturing the image of the corresponding viewer. Each camera may be placed above, below, or to either lateral side of the associated display. In the illustrative prior art system shown in FIG. 1, cameras 16 and 18, respectively, are placed directly above displays 12 and 14. The angle ~ represents the angle between camera 16 and display 12 relative to 30 viewer A. Because participant A is looking directly at the displayed image of participant B, and therefore not at camera 16, the displayed image of A appears to be looking down.
The parallax effect has a similar impact on the displayed image of participant B.
As will be readily ascertained by those skilled in the art, the parallax effect can be minimi7Pd by placing the camera and monitor as close together as possible. In fact, 3s previous efforts to elimin~e the parallax effect have relied upon positioning the camera directly behind a specially constructed display screen. In U.S. Patent No. 4,054,908 issued to Poieier et al on October 18, 1977, for example, the image of the observer is reflected to the camera lens by an oblique reflecting plate. The display screen is viewed by the observer through a window in the reflecting plate. The presence of the window in the apertured plate, however, is somewhat distracting to the viewer. It has also been proposed to utilize an electronic crystal screen which alternates between opaque and transparent states at a frequency slot enough to permit an image of the viewer to reach the camera while in the transparent state and to present a displayed image while in the opaque, reflecting state. Disadvantageously, images displayed on such a flashing screen are characterized by unacceptably low brightness. Moreover, the screen structure andassociated driving circuitry are costly to fabricate.
So-called virtual space video conferencing systems have been proposed in an effort to provide full visual presence and convey the spatial relationship between three or more conference participants. In this type of system, the number of monitor and camera pairs at each site corresponds to the total number of remote conference participants, and the 1S relative positions of the pairs relative to each participant are fixed at each site. An illustrative system of this type is depicted in FIG. 2. Each viewing subject (A,B,C, and D) sits in a swivel chair in a corresponding room ( 1,2,3, or 4) with three TV monitors representing the other three "virtually present" participants. Thus, for example, the viewing subject (A) sits in a swivel chair in Room 1 with three TV monitors representing the three "virtually present" participants (B, C, and D). The arrangement of monitor/camera pairs in each room duplicates the positions of each participant as if they were all seated at the same square table. The signals from these cameras pass through video switching units and are directed to corresponding destinations monitors by a dynamic allocation network.
2s The virtual space system of FIG. 2 provides three or more participants with the ability to see who is looking at whom during a conference. Eye contact between the participants, however, has heretofore been obtained by resorting to the apertured or semi-reflecting screen devices discussed above. Moreover, the large number of video channels needed to provide the desired spatial distribution between the conference participants translates into a large expenditure of transmission bandwidth.
Summar~ of the Invention The system and process of the present invention overcomes the aforementioned disadvantages by using real-time image analysis and processing techniques to provide me~ningful head orientation and/or eye contact in the images presented to the remote 3s viewer. A video conferencing system constructed in accordance with the present invention includes, in each conference room, at least one video telephone that include a video camera for generating video signals indicative of a sequence of local conferee images frames, and an image receiver for displaying image frames of at least one remote conferee. The image receiver, the video camera, and the eyes of the local conferee define a parallax angle. The video conference system further includes a frame generating system, 5 responsive to the video signals, for analyzing local conferee image frames and generating a corresponding sequence of parallax compensated frames. A signal indicative of each respective sequence of parallax compensated frames is tr~nsmitted to a corresponding image receiver, whereby eye contact is provided between each local conferee and a displayed image of a corresponding remote conferee.
If more than two conferees are involved, each input image frame is analyzed for head position and reoriented, as necessary, by the frame generating means to provide a sense of presence. Illustratively, each image receiver may present two remote conferees on a split window display. Each input image frame is mapped onto a three dimensional ellipsoid head model, reoriented by a predetermined angle to obtain the proper gaze lS direction, and recapped onto a two dimensional reconstructed image frame. A signal representing the reconstructed image frames is then presented to the appr~,pliate remote image receivers. Thus, in accordance with the present invention, it is possible for each conferee to discern who is looking at whom during the conference.
Other features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims.
Brief Description of the Drawin~
2s In the drawings, wherein like reference characters denote similar elements throughout the several views:
FIG. 1 depicts the operation of a conventional single person camera (SPC) video conference system;
FIG. 2 depicts a conventional virtual space video conferencing system;
F~G. 3A is a block diagram depicting a video conferencing system constructed in accordance with an illustrative embodiment of the present invention;
FIG. 3B is an elevational view depicting the video conferencing system of FIG.
3A;
F~G. 4 is a graphical representation of a look up table for providing pixel values in accordance with an eye synthesis technique utilized by the present invention;
4 ~lS7613 FIG. S is a flowchart depicting the steps of an inventive image analysis and processing technique by which parallax correction is provided in accordance with the present invention;
FIG. 6 is an elevational view illustrating the operation of a video conferencings system providing a sense of presence between three participants in accordance with the invention; and FIG. 7 is a flowchart depicting the steps of a modified image analysis and processing technique by which parallax correction and head reorientation are performed to reproduce a sense of presence during operation of the system of FIG. 5.
0 Detailed Descfiption A conferencing system 20 constructed in accordance with an illustrative embodiment of the present invention is depicted in FIG. 3A. As soon in FM. 3A, each participant in a video teleconference views a respective display 22, 24. Each display may be divided into one or more windows for presenting the image of a corresponding number 15 of remote participants. Illustratively, there by be two conference sites with one conference participant at each site. Thus, in FIG. 3A, participants A and B are shown both a corresponding conference sites 26, 28 and as they appear on displays 22 and 24, respectively.
In the illustrative embodiment depicted in FIG. 3A, the images of conference 20 participants A and B are initially obtained by video cameras 30 and 32, respectively. In a conventional manner, the analog output signals of cameras 30 and 32 are converted to a digital form, pre-processed, and encoded for bit rate reduction to facilitate tr~ncmi~ion over or via a telecommunication network (not shown). Systems and techniques for performing the aforementioned signal processing functions are well known and are not 2s deemed to constitute a novel aspect of the present invention; accordingly, a detailed description of the same is believed unnPces.s:~ry and has been omitted.
With continued reference to FIG. 3A, it can be seen that the digitized video signals from each camera are not tr~nsmitted directly to the associated destination display.
Rather, and for a purpose which will now be explained, the outputs of cameras 30 and 32 30 are supplied to a suitable processing device 34, such for example as a high processing speed computer. Essentially, processing device 34 provides frame by frame analysis, in real time, of the raw incoming video that that it receives and modifies the image of each incoming frame so as to restore eye contact between the displayed image and the observer.
The modified frames are then tr~n~mitted, in real time, to the appropriate destination 35 display.
As was discussed above with reference to FIG. 1, the positioning of the camera directly above the display results in the formation of a parallax angle ~ between the camera, the eyes of observer A, and the displayed image of participant B. This arrangement of camera, display, and observer is repeated in FIG. 3B. Although each 5 camera is shown as being positioned above its corresponding display, it will be readily appreciated by those skilled in the art that the cameras may be positioned below or at either lateral side of the display, as well. In accordance with the present invention, parallax correction is performed by procescing device 34. For this purpose, it is first necessary to determine the parallax angle ~.
In order of increasing precision, there are several ways to determine the parallax angle ~. In the illustrative embodiment of FIGS. 3A and 3B, the images of only two conference participants, looking straight ahead, need be modified. Assuming the movements of A and B are negligible, a fixed angle ~ can be determined based on the relative positions of the camera and the display. ~lttqrn~tively, if it is assumed that the 15 image of B on display A does not move very much, then the image taken by camera A may be analyzed to determine the amount of correction needed for A's image to face straight ahead. The amount of correction required, denoted as ~, can be modeled as a random variable with a certain probability distribution, stationary or dme varying. With the assumption that A is looking at the image of B most of the time, the parallax angle ~ can 20 be equated with the expectation value of ~, i.e. E[~].
When employing either of the above-mentioned lack-of-movement assumptions, it is only n~cess:~ry to locate the eyes in each image frame and to modify the frames to correct for the parallax angle ~. Feature extraction techniques for locating facial features such as the eyes, nose and mouth in a digitized image are well known and are not2s described in detail here. Similarly, techniques for modifying localized areas of such images are known to those skilled in this art. In any event, because the eyes occupy only a small portion of the entire face or image thereof, the task of turning the eyes forward can be done by simply synthesizing the eyes. The feasibility of eye synthesis relies on the natural tolerance of the human viewer to mirror noise in the relatively small region of the eyes. In 30 this regard, it should be noted that the position of the camera relative to the display affects the complexity of the eye synthesis algorithm as well as the quality of the results obtained.
Specifically, the eye lid positions are significantly affected by the vertical component of the gaze direction, so that if the eyes had to be moved vertically, as would those in the displayed image of G, the lid would also have to be altered to avoid perceptual 3s incongruity. With the camera to the side of the display, lid position does not have to change as the eye is synthesized. Accordingly, a camera position to the side of the display is especially preferred.
In order to synthPsi7P an image of the eye, the program must determine the boundary of the surrounding lids. The lids primarily adjoin the white, but often also S adjoining the iris at the top and bottom. The entire region within this boundary is synthesized. If the subject is far away from the camera (so that the eyes in the obtained image are very small), the position of the pupil needed to provide eye contact with the observer can be established merely by adding a properly size, dark circular region at the corrected position.
If the subject is sufficiently close to the camera/monitor, completely new intensity values may be created for each pixel position within the outline of the eye, that is, between the upper and lower lids. In addition to the eye' s outline, five additional parameters are utilized to synthesiæ a new eye: the two coordinates of the pupil's center, the iris radius, the iris intensity level, and the white's intensity level. A boundary detection method 15 naturally results in a simple boundary representation containing the minimum and maximum iris X coordinates for each Y value within a range. The iris intensity is computed during the basic eye tracking algorithm, and similarly the white intensity and eye outline are computed during the eye white boundary deterrnin~tion. Illustratively, eye synthesis may be performed in accordance with a parametric mixing function approach in 20 which the iris diameter is proportional to the eye-to-eye distance. The eye-to-eye distance provides the proper scaling as the subject moves closer to and farther away from the camera.
As discussed above, a fixed angle ~ can be determined based on the relative positions of the camera and the display. Illustratively, the camera may be positioned to 25 one side of the display and at the same height. For purposes of the present invention, it may be assumed that the user is looking at the display and not somewhere else. Thus, if the user looks somewhere else, the synthe~i7ed image no longer appears to m~int~in eye contact with the viewer. With the camera and display arranged in this manner, the iris and pupil may be shifted horizontally by a certain geometry-dependent amount, with no 30 vertical motion. The magnitude of the shift is inversely proportional to the eye-to-eye distance. Illustratively, the synthesi7ing algorithm may be configured to compute the distance to the synth~.ciæd pupil center, in aspect-ratio-corrected horizontal pixels. A
look up table, based on the ratio of this distance to the synthesized iris radius, supplies a mixing coefficient between the iris level and the white level: 1 specifies the iris level and 0 35 specifies the white level, with linear mixing in between. A graphical representation of the -look-up table is shown in FM. 4. The entries above 1 produce an even darker level for the pupil. The synthesis of the white may utilize a similar process.
As will be readily appreciated by those skilled in the art, an eye image synthe~i~Pd in the above described manner will lack the reflected highlight which gives the eye an s impression of depth and contour. As such, it may be desirable to add such highlights to the image in a conventional manner. Utilizing a Motorola 88000 RISC processor toperform the aforementioned image analysis and processing operations on a single input video frame, it took 1.177 msec to locate the pupil and white boundaries, 0.176 msec to post process the boundaries, 1.670 msec to synthesize the new eyes, and 1.453 msec to 0 generate real-time graphics and text displays. Of course, the processor times are sensitive to the eye's image in the image plane. However, even with extreme close-ups, it is possible to generate a modified frame having synthesiæd eyes within one frame interval.
As a result, a sequence of parallax compensated image frame may be generated in real-time to advantageously provide the users of a video conferencing system with eye contact S without sacrifices in frame rate.
It may not be possible to utilize the aforementioned assumptions as to the movements of A and B. If compensation for the exact parallax angle ~ is required, it is necessary to detect A's eye position and B's eye position on display 24. Any suitable image analysis technique may be utilized to detect the eye positions for the purposes of the 20 present invention. By way of illustrative example only, a non-intrusive gaze tracking technique that may be utilized to obtain eye position information is described by S. Baluja and D. A. Pomerleau in an article entitled "Non-Intrusive Gaze Tracking Using Artificial Neural Networks", 6 Advances in Neural Information Processing Systems (NIPS) (1994).
It should be noted that a non-intrusive gaze technique is preferred so that each conference 2s participant may move his or her head freely.
Essentially, the goal of gaze tracking is to determine where a subject is looking from the appearance of the subject's eye. In order to account for shifts in the relative positions of the camera and the eye, the eye is first located in each image frame.
Typically, one of the eyes is located by searching for the specular reflection of a stationary 30 light source in the image of the subject's face. Although any suitable light source may be utilized for the purposes of the present invention, a non-intrusive infrared source is preferred since it provides an easily detected reflection pattern without creating a distraction or otherwise causing discomfort to the viewer. The specular reflection in the image can usually be distinguished by a small bright region surrounded by a very dark 3s region. The reflection location may be utilized to limit the search fro the eye in the next frame. A window surrounding the reflection is extracted and the image of the eye is located within this window. The center of the pupil is found and the relative position of the light's reflection to the pupil's center is calculated. The gaze direction is determined from information about the relative positions. For example, when the observer is looking at the light source, the specular reflection is at the center of the pupil image. If the observer is looking above the light, the specular reflection is near the bottom edge of the pupil image. If the observer is looking to the left of the light, the specular reflection is near the right of the pupil image, and so on.
In the illustrative system depicted in FIGS. 3A and 3B, a fixed light source 36 is positioned above camera 32 to facilitate the use of a gaze tracking technique on the image of participant A. A similar light source (not shown) is positioned above camera 30. As will be readily appreciated by those skilled in the art, a gaze detection technique such as the one discussed above provides useful information concerning the direction of the eye relative to the light source. As will not be explained, this information may be used to determine the parallax angle ~. Specifically, and as best seen in FIG. 3B, light source 36, participant A, and the displayed image of participant B define angle ~317 while the light source 36, participant A, and camera 32 define angle 2- Parallax angle 0 may thus be calculated by taking the difference between l and 2- Once the parallax angle is known, generation and tr~nsmiccion of a revised image frame is performed in the same manner as described above.
An illustrative sequence of steps for performing parallax correction with a video conferencing system such as that depicted in FIGS. 3A and 3B is shown in FIG. 5.Although the discussion will proceed in connection with the processing of images of only one participant, it will be readily apparent that the same image analysis and processing steps are simultaneously performed for all conferee participants. In any event, a digitized video frame of participant A is received by processing device 34 (step 40) and image feature extraction is performed to locate the eyes in the image (step 42). During step 44, gaze detection is performed on the image to detect the direction in which the subject is looking relative to the camera. The parallax angle is derived (step 46) and a revised image frame is generated (step 48) in which the positions of the pupils and, if appropliate, the irises correspond with the positions need for eye contact with observing participant B.
The revised image frame is then tr~ncmitted to display 22 (step 50) where it is viewed by participant B. The aforementioned steps are performed, in real time and frame by frame for each video signal input, so that eye contact is provided between the displayed images and the respective participants.
When more than two conference sites and/or participants are involved, additionalconsiderations besides the need to provide eye contact between the participants must be addressed in order for a video conferencing system to preserve the look and feel of a live meeting. Specifically, and as already discussed above, it is also important for each participant to see who is talking to whom. Thus, in accordance with the teachings of the present invention, real-time image analysis and processing techniques are utilized to s reorient the head image as well as to provide the conference participants with the appearance of eye contact.
FIG. 6 again depicts a video conferencing system 60 constructed in accordance with the present invention and similar to the system 20 of FIG. 3A. The system illustrated in FIG. 6, however, has been modified to provide a conference between three participants 10 A, B, and C at three different sites. System 60 therefore includes three respective displays 62, 64, and 66, and three corresponding cameras 68, 70, and 72. A light source, only one of which is shown and indicated generally at 76, is positioned above each camera Each of the displays are divided into two windows, with each window presenting the image of two remote participants. In a manner identical to that discussed above, the video signals from 5 each camera are digitized, processed and transmitted to a processing device such as s the processing device 74.
At the point of the video teleconference depicted in FIG. 6, A is speaking to B. In an actual conference, A would be looking at B, and both B and C would be looking at A.
Such each participant, absent the additional image processing of the invention, is facing 20 straight ahead at his or her display, however, the respective images presented to the other participants will not preserve the corresponding head orientations. To obtain the proper orientations depicted in FIG. 6, varying amounts of image analysis and processing must be performed, as will now be described.
While A is looking at the window of B on display 62, A should see the images of B
2s and C facing straight ahead (i.e. facing A). Since B and C are already facing their own cameras, only parallax error correction of their eye positions is required. Such error correcdon may be performed utilizing any of the techniques described above in connecdon with FIGS. 3A and 3B. On display 64, B should see the image of A facing straight ahead, and the image of C facing the image of A. While only parallax error correction of A's 30 image is required for display 64, the image of C additionally requires reorientation to achieve the position shown in FIG. 6. Similarly, on display 66, C should see A and B
facing each other. Thus, the images of both A and B require at least head reorientation to achieve the posidons depicted in FIG. 6.
A process for performing head reorientation and parallax correction in accordance 35 with an illustrative embodiment of the present invention is depicted in the flow chart of FIG. 7. A digidzed video frame of a conference participant such, for example, as lo 2157613 participant C is received by processing device 74 (step 80) and image feature extraction is performed to locate the eyes in the image (step 82). During step 84, gaze detection is performed on the image to detect the direction in which the subject is looking relative to the camera. When participant C is looking at participant A, for example, the specular s reflection of light source 76, in the image of participant C's eye as obtained by camera 72, will be above and to the right of the center of the pupil.
As discussed above, the parallax angle 0 can be computed from this information given the fixed position of camera 72 and display 66 (step 86). More significantly, however, the position of the specular reflection relative to the pupil center may be utilized to detect which image is being watched by the observing participant (step 88). In the present example, the location of the specular reflection to the right of the pupil center in the received image of participant C suggests that C is looking to the left, and therefore at participant A.
Once it is known which display window is being viewed by each conference participant, image processing is performed to generate respective, corrected image frames that are apprupliate for each destination display. If only parallax correction is required, as determined at step 88, a first frame is generated for transmission to the applopflate display(s) (step 90). In the illustrative example depicted in FIG. 6, a frame indicative of the parallax corrected, forward-facing image of C is presented to display 62. Because reorientation of C's head is required to convey a sense of presence to participant B, however, several additional proces~ing steps must now be performed.
The face image of each frame is texture mapped onto a three dimensional (3-D) head model (step 92). Illustratively, a relatively simple ellipsoid head model may be employed for this purpose. Of course, it will be readily appreciated by those skilled in the 2s art that other, more sophisticated 3-D models, may also be employed, depending upon the processing power and speed of execution limitations imposed by the specific application.
By way of additional example, a wire frame model as disclosed by K. Aizawa, H.
Harashima, and T. Saito in a paper entitled "Model-based Analysis Synthesis Image Coding System for a Person's Face", Signal Processing Image Communication, Vol. 1, No. 2, October 1989, pp. 139-152, may be utilized. In any event, the resulting three-dimensional, textured model is rotated in the appropriate direction by a predetermined angle a (step 94) and the texture of the ellipsoid model is projected back to the 2D
viewing plane to produce the revised image (step 96). The thus generated image is then transmitted to the appropriate destination display(s), as shown in step 98. In the 3s illustrative example depicted in FIG. 6, the image of C presented to display 64 has been rotated 90 to the right to face in the direction of the displayed image of participant A.
The eyeballs of image C can be independently modeled in a similar manner or synthesi7ed in connection with the pure parallax correction technique discussed above.
Along with video coding, parallax correction and/or head orientation can be performed at the respective conference sites as preprocessing or post-processing. In s either case, image analysis is preferably performed before coding of the original image signals so that no critical image information is lost prior to analysis. Synthesis of images corrected in accordance with the present invention, however, may be performed anywhere.
In the pre-processing case, the image is digitized, analyzed, synthesi7ed/processed, coded, and tr~n~mi~ted. Because different processed images must be sent to different 10 participants, a large amount of tr~n~mic.cion bandwidth is required. In the post-processing case, the parallax angle and head orientation information of each participant obtained by image analysis must be sent along with the coded video signals. Synthesis may then be performed at the receiver end, or in some centralized location.
While there have been shown and described and pointed out fundamental novel 5 features of the invention as applied to preferred embodiments thereof, it will be understood that various omissions and substitutions and changes in the form and details of the disclosed invention may be made by those skilled in the art without departing from the spirit of the invention. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.
Claims (10)
1. A method for establishing a video conference between a plurality of remotely located conference rooms each containing at least one conferee, a video camera, and an image receiver, the image receiver, video camera, and eyes of each conferee associated therewith defining a parallax angle of said each conferee, characterized by the steps of:
generating, with a first video camera, a video signal representative of a sequence of input image frames of a first conferee;
analyzing said input image frames of a first conferee, in real time, to determine a parallax angle 8 in each frame;
generating a corresponding sequence of parallax-compensated image frames;
and transmitting a signal indicative of said sequence of parallax-compensated image frames to an image receiver viewed by a second conferee, whereby eye contact is provided between said second conferee and a displayed image of said first conferee.
generating, with a first video camera, a video signal representative of a sequence of input image frames of a first conferee;
analyzing said input image frames of a first conferee, in real time, to determine a parallax angle 8 in each frame;
generating a corresponding sequence of parallax-compensated image frames;
and transmitting a signal indicative of said sequence of parallax-compensated image frames to an image receiver viewed by a second conferee, whereby eye contact is provided between said second conferee and a displayed image of said first conferee.
2. The method according to claim 1, characterized in that said analyzing step includes locating, in an image of said first conferee, a specular reflection of a fixed light source in a pupil of the first conferee.
3. The method according to claim 2, characterized in that said analyzing step further includes measuring a position of said specular reflection relative to the center of said pupil.
4. The method according to claim 1, characterized in that said generating step includes synthesizing at least a portion of each parallax-compensated image frame from an input conferee image frame.
5. The method according to claim 1, characterized in that a gaze direction of said first conferee is also obtained during said measuring step and wherein said generating step further includes texture mapping each input conferee image frame onto a three dimensional ellipsoid facial model and reorienting said facial model to achieve a facial orientation selected in accordance with said gaze direction.
6. A video conference system connecting a plurality of remotely located conference rooms, each room containing at least one of a plurality of conferees, characterized by:
at least one video telephone in each room comprising a video camera for generating video signals indicative of a sequence of image frames of a first conferee and an image receiver for displaying image frames of at least a second conferee, said image receiver, said video camera, and eyes of said first conferee defining a parallax angle .THETA.;
frame generating means, responsive to said video signals, for analyzing input image frames of said first conferee and for generating a corresponding sequence of parallax-compensated image frames of said first conferee; and means for transmitting a signal indicative of said sequence of parallax-compensated frames to at least one remote image receiver, whereby eye contact is provided between at least said second conferee and a displayed image of said first conferee.
at least one video telephone in each room comprising a video camera for generating video signals indicative of a sequence of image frames of a first conferee and an image receiver for displaying image frames of at least a second conferee, said image receiver, said video camera, and eyes of said first conferee defining a parallax angle .THETA.;
frame generating means, responsive to said video signals, for analyzing input image frames of said first conferee and for generating a corresponding sequence of parallax-compensated image frames of said first conferee; and means for transmitting a signal indicative of said sequence of parallax-compensated frames to at least one remote image receiver, whereby eye contact is provided between at least said second conferee and a displayed image of said first conferee.
7. The system according to claim 6, characterized by a fixed light source in each room for producing a specular reflection on the eyes of the conferee in said each room.
8. The system according to claim 7, characterized in that said frame generating means is operative to analyze said input conferee image frames and detect a gaze direction of said first conferee.
9. The system according to claim 8, characterized in that said frame generating means is operative to generate a second image frame for transmission to at least a third conferee, said frame generating means being responsive to a detected gaze direction to texture map each input conferee image frame onto a three dimensional ellipsoid facial model and reorient said facial model to achieve a facial orientation selected in accordance with said detected gaze direction.
10. The system according to claim 9, characterized in that said transmitting means is operable to transmit a first sequence of parallax corrected image frames representative of said first conferee to an image receiver positioned proximate a second conferee and to transmit a second sequence of gaze direction-compensated image frames representative of said first conferee to an image receiver positioned proximate a third conferee, thereby providing a sense of presence between said first, second, and third conferees.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US08/328,640 US5500671A (en) | 1994-10-25 | 1994-10-25 | Video conference system and method of providing parallax correction and a sense of presence |
| US328,640 | 1994-10-25 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2157613A1 CA2157613A1 (en) | 1996-04-26 |
| CA2157613C true CA2157613C (en) | 1999-09-21 |
Family
ID=23281793
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002157613A Expired - Fee Related CA2157613C (en) | 1994-10-25 | 1995-09-06 | Video conference system and method of providing parallax correction and a sense of presence |
Country Status (6)
| Country | Link |
|---|---|
| US (1) | US5500671A (en) |
| JP (1) | JPH08237629A (en) |
| CA (1) | CA2157613C (en) |
| DE (1) | DE19539048B4 (en) |
| GB (1) | GB2294605B (en) |
| TW (1) | TW297985B (en) |
Families Citing this family (227)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8352400B2 (en) | 1991-12-23 | 2013-01-08 | Hoffberg Steven M | Adaptive pattern recognition based controller apparatus and method and human-factored interface therefore |
| US10361802B1 (en) | 1999-02-01 | 2019-07-23 | Blanding Hovenweep, Llc | Adaptive pattern recognition based control system and method |
| JPH07222202A (en) * | 1994-02-02 | 1995-08-18 | Rohm Co Ltd | Stereoscopic vision camera |
| USRE38884E1 (en) * | 1995-02-16 | 2005-11-22 | Sumitomo Electric Industries, Ltd. | Two-way interactive system, terminal equipment and image pickup apparatus having mechanism for matching lines of sight between interlocutors through transmission means |
| US5815197A (en) * | 1995-02-16 | 1998-09-29 | Sumitomo Electric Industries, Ltd. | Two-way interactive system, terminal equipment and image pickup apparatus having mechanism for matching lines of sight between interlocutors through transmission means |
| JPH08256318A (en) * | 1995-03-17 | 1996-10-01 | Fujitsu Ltd | Camera controller for video conference system |
| KR100414629B1 (en) * | 1995-03-29 | 2004-05-03 | 산요덴키가부시키가이샤 | 3D display image generation method, image processing method using depth information, depth information generation method |
| US6020892A (en) * | 1995-04-17 | 2000-02-01 | Dillon; Kelly | Process for producing and controlling animated facial representations |
| JP3316725B2 (en) * | 1995-07-06 | 2002-08-19 | 三菱電機株式会社 | Face image pickup device |
| US5559875A (en) * | 1995-07-31 | 1996-09-24 | Latitude Communications | Method and apparatus for recording and retrieval of audio conferences |
| US5675376A (en) * | 1995-12-21 | 1997-10-07 | Lucent Technologies Inc. | Method for achieving eye-to-eye contact in a video-conferencing system |
| US5917940A (en) * | 1996-01-23 | 1999-06-29 | Nec Corporation | Three dimensional reference image segmenting method and device and object discrimination system |
| US6037970A (en) * | 1996-04-05 | 2000-03-14 | Sony Corporation | Videoconference system and method therefor |
| US5856842A (en) * | 1996-08-26 | 1999-01-05 | Kaiser Optical Systems Corporation | Apparatus facilitating eye-contact video communications |
| US6275258B1 (en) | 1996-12-17 | 2001-08-14 | Nicholas Chim | Voice responsive image tracking system |
| US6259470B1 (en) * | 1997-12-18 | 2001-07-10 | Intel Corporation | Image capture system having virtual camera |
| US6593956B1 (en) | 1998-05-15 | 2003-07-15 | Polycom, Inc. | Locating an audio source |
| EP1039752A4 (en) * | 1998-10-09 | 2007-05-02 | Sony Corp | Communication apparatus and method |
| US6393136B1 (en) * | 1999-01-04 | 2002-05-21 | International Business Machines Corporation | Method and apparatus for determining eye contact |
| US7966078B2 (en) | 1999-02-01 | 2011-06-21 | Steven Hoffberg | Network media appliance system and method |
| US7117157B1 (en) * | 1999-03-26 | 2006-10-03 | Canon Kabushiki Kaisha | Processing apparatus for determining which person in a group is speaking |
| GB9908545D0 (en) | 1999-04-14 | 1999-06-09 | Canon Kk | Image processing apparatus |
| KR100307854B1 (en) * | 1999-11-10 | 2001-11-02 | 전병우 | Method and Apparatus of Gaze Compensation for Eye Contact Using Single Camera |
| US6806898B1 (en) * | 2000-03-20 | 2004-10-19 | Microsoft Corp. | System and method for automatically adjusting gaze and head orientation for video conferencing |
| WO2001082154A1 (en) * | 2000-04-21 | 2001-11-01 | Shiseido Company, Ltd. | Makeup counseling apparatus |
| WO2001084838A1 (en) * | 2000-04-28 | 2001-11-08 | Swisscom Mobile Ag | Method and system for video conferences |
| US20020015003A1 (en) * | 2000-08-07 | 2002-02-07 | Masami Kato | Virtual space system structured by plural user terminals and server device |
| JP3784289B2 (en) * | 2000-09-12 | 2006-06-07 | 松下電器産業株式会社 | Media editing method and apparatus |
| US6718051B1 (en) | 2000-10-16 | 2004-04-06 | Xerox Corporation | Red-eye detection method |
| US6724417B1 (en) | 2000-11-29 | 2004-04-20 | Applied Minds, Inc. | Method and apparatus maintaining eye contact in video delivery systems using view morphing |
| US6785402B2 (en) * | 2001-02-15 | 2004-08-31 | Hewlett-Packard Development Company, L.P. | Head tracking and color video acquisition via near infrared luminance keying |
| US6714234B1 (en) | 2001-04-11 | 2004-03-30 | Applied Minds, Inc. | Maintaining eye-contact in teleconferencing using structured light |
| WO2002084590A1 (en) * | 2001-04-11 | 2002-10-24 | Applied Minds, Inc. | Knowledge web |
| US20030107645A1 (en) * | 2001-08-17 | 2003-06-12 | Byoungyi Yoon | Method and system for controlling the display location of stereoscopic images |
| US6798457B2 (en) | 2001-09-26 | 2004-09-28 | Digeo, Inc. | Camera positioning system and method for eye-to-eye communication |
| US6943843B2 (en) * | 2001-09-27 | 2005-09-13 | Digeo, Inc. | Camera positioning system and method for eye-to eye communication |
| US20030169339A1 (en) * | 2001-10-01 | 2003-09-11 | Digeo. Inc. | System and method for tracking an object during video communication |
| US6598971B2 (en) | 2001-11-08 | 2003-07-29 | Lc Technologies, Inc. | Method and system for accommodating pupil non-concentricity in eyetracker systems |
| US20040207718A1 (en) * | 2001-11-14 | 2004-10-21 | Boyden James H. | Camera positioning system and method for eye -to-eye communication |
| US7130446B2 (en) * | 2001-12-03 | 2006-10-31 | Microsoft Corporation | Automatic detection and tracking of multiple individuals using multiple cues |
| US20030112325A1 (en) * | 2001-12-13 | 2003-06-19 | Digeo, Inc. | Camera positioning system and method for eye-to-eye communication |
| US7126627B1 (en) | 2002-03-06 | 2006-10-24 | Lewis Thomas B | Video conferencing device and method |
| US7904826B2 (en) * | 2002-03-29 | 2011-03-08 | Microsoft Corporation | Peek around user interface |
| US20030195834A1 (en) * | 2002-04-10 | 2003-10-16 | Hillis W. Daniel | Automated online purchasing system |
| US8069175B2 (en) | 2002-04-10 | 2011-11-29 | Google Inc. | Delegating authority to evaluate content |
| US7844610B2 (en) | 2003-12-12 | 2010-11-30 | Google Inc. | Delegated authority evaluation system |
| EP1365359A1 (en) * | 2002-05-24 | 2003-11-26 | BRITISH TELECOMMUNICATIONS public limited company | Image processing method and system |
| US8012025B2 (en) * | 2002-12-13 | 2011-09-06 | Applied Minds, Llc | Video game controller hub with control input reduction and combination schemes |
| AU2003297193A1 (en) | 2002-12-13 | 2004-07-09 | Applied Minds, Inc. | Meta-web |
| US7106358B2 (en) * | 2002-12-30 | 2006-09-12 | Motorola, Inc. | Method, system and apparatus for telepresence communications |
| US7593546B2 (en) * | 2003-03-11 | 2009-09-22 | Hewlett-Packard Development Company, L.P. | Telepresence system with simultaneous automatic preservation of user height, perspective, and vertical gaze |
| US8299979B2 (en) * | 2003-05-14 | 2012-10-30 | Broadcom Corporation | Integral eye-path alignment on telephony and computer video devices using two or more image sensing devices |
| US8363951B2 (en) * | 2007-03-05 | 2013-01-29 | DigitalOptics Corporation Europe Limited | Face recognition training method and apparatus |
| US8553949B2 (en) * | 2004-01-22 | 2013-10-08 | DigitalOptics Corporation Europe Limited | Classification and organization of consumer digital images using workflow, and face detection and recognition |
| GB0321083D0 (en) * | 2003-09-09 | 2003-10-08 | British Telecomm | Video communications method and system |
| US7336296B2 (en) * | 2003-10-10 | 2008-02-26 | International Business Machines Corporation | System and method for providing position-independent pose estimation |
| US20050131918A1 (en) * | 2003-12-12 | 2005-06-16 | W. Daniel Hillis | Personalized profile for evaluating content |
| US8824730B2 (en) * | 2004-01-09 | 2014-09-02 | Hewlett-Packard Development Company, L.P. | System and method for control of video bandwidth based on pose of a person |
| US7564994B1 (en) * | 2004-01-22 | 2009-07-21 | Fotonation Vision Limited | Classification system for consumer digital images using automatic workflow and face detection and recognition |
| US7865834B1 (en) * | 2004-06-25 | 2011-01-04 | Apple Inc. | Multi-way video conferencing user interface |
| EP1613082A1 (en) * | 2004-06-30 | 2006-01-04 | Sony Ericsson Mobile Communications AB | Face image correction |
| US8503800B2 (en) | 2007-03-05 | 2013-08-06 | DigitalOptics Corporation Europe Limited | Illumination detection using classifier chains |
| US7715597B2 (en) * | 2004-12-29 | 2010-05-11 | Fotonation Ireland Limited | Method and component for image recognition |
| US7605837B2 (en) * | 2005-06-02 | 2009-10-20 | Lao Chan Yuen | Display system and method |
| CN101496387B (en) | 2006-03-06 | 2012-09-05 | 思科技术公司 | System and method for access authentication in a mobile wireless network |
| GB0718706D0 (en) | 2007-09-25 | 2007-11-07 | Creative Physics Ltd | Method and apparatus for reducing laser speckle |
| US7591558B2 (en) * | 2006-05-31 | 2009-09-22 | Sony Ericsson Mobile Communications Ab | Display based on eye information |
| US20070279483A1 (en) * | 2006-05-31 | 2007-12-06 | Beers Ted W | Blended Space For Aligning Video Streams |
| KR100820639B1 (en) * | 2006-07-25 | 2008-04-10 | 한국과학기술연구원 | System and method for 3-dimensional interaction based on gaze and system and method for tracking 3-dimensional gaze |
| JP4730667B2 (en) * | 2006-07-25 | 2011-07-20 | 富士フイルム株式会社 | Automatic regeneration method and apparatus |
| US7515740B2 (en) * | 2006-08-02 | 2009-04-07 | Fotonation Vision Limited | Face recognition with combined PCA-based datasets |
| US8130261B2 (en) * | 2006-10-10 | 2012-03-06 | Exelis, Inc. | System and method for dynamically correcting parallax in head borne video systems |
| US8648897B2 (en) * | 2006-10-10 | 2014-02-11 | Exelis, Inc. | System and method for dynamically enhancing depth perception in head borne video systems |
| US20080117290A1 (en) * | 2006-10-18 | 2008-05-22 | Mgc Works, Inc. | Apparatus, system and method for generating stereoscopic images and correcting for vertical parallax |
| AU2007221976B2 (en) * | 2006-10-19 | 2009-12-24 | Polycom, Inc. | Ultrasonic camera tracking system and associated methods |
| FR2910770A1 (en) * | 2006-12-22 | 2008-06-27 | France Telecom | Videoconference device for e.g. TV, has light source illuminating eyes of local user viewing screen during communication with remote user, such that image sensor captures local user's image with reflection of light source on eyes |
| EP2123008A4 (en) * | 2007-03-05 | 2011-03-16 | Tessera Tech Ireland Ltd | Face categorization and annotation of a mobile phone contact list |
| US20080278516A1 (en) * | 2007-05-11 | 2008-11-13 | Santon John C | System and method for adjusting perceived eye rotation in image of face |
| US8570373B2 (en) | 2007-06-08 | 2013-10-29 | Cisco Technology, Inc. | Tracking an object utilizing location information associated with a wireless device |
| GB2450345A (en) * | 2007-06-20 | 2008-12-24 | Iona Simpson | Videoconference terminal providing enhanced user interface |
| US8074581B2 (en) | 2007-10-12 | 2011-12-13 | Steelcase Inc. | Conference table assembly |
| US20090183276A1 (en) * | 2008-01-10 | 2009-07-16 | Pioneer Hi-Bred International, Inc. | Novel Bacillus Thuringiensis Gene with Coleopteran Activity |
| US8355041B2 (en) | 2008-02-14 | 2013-01-15 | Cisco Technology, Inc. | Telepresence system for 360 degree video conferencing |
| US8797377B2 (en) | 2008-02-14 | 2014-08-05 | Cisco Technology, Inc. | Method and system for videoconference configuration |
| US8289367B2 (en) * | 2008-03-17 | 2012-10-16 | Cisco Technology, Inc. | Conferencing and stage display of distributed conference participants |
| US8319819B2 (en) | 2008-03-26 | 2012-11-27 | Cisco Technology, Inc. | Virtual round-table videoconference |
| US8390667B2 (en) | 2008-04-15 | 2013-03-05 | Cisco Technology, Inc. | Pop-up PIP for people not in picture |
| NO331839B1 (en) * | 2008-05-30 | 2012-04-16 | Cisco Systems Int Sarl | Procedure for displaying an image on a display |
| US20100054526A1 (en) * | 2008-09-03 | 2010-03-04 | Dean Eckles | Method, apparatus and computer program product for providing gaze information |
| US8694658B2 (en) | 2008-09-19 | 2014-04-08 | Cisco Technology, Inc. | System and method for enabling communication sessions in a network environment |
| US8593503B2 (en) * | 2008-09-25 | 2013-11-26 | Alcatel Lucent | Videoconferencing terminal and method of operation thereof to maintain eye contact |
| US10631632B2 (en) | 2008-10-13 | 2020-04-28 | Steelcase Inc. | Egalitarian control apparatus and method for sharing information in a collaborative workspace |
| US20140361954A1 (en) | 2013-06-07 | 2014-12-11 | Lewis Epstein | Personal control apparatus and method for sharing information in a collaboration workspace |
| US8266536B2 (en) * | 2008-11-20 | 2012-09-11 | Palo Alto Research Center Incorporated | Physical-virtual environment interface |
| US8405706B2 (en) * | 2008-12-17 | 2013-03-26 | Microsoft Corporation | Visual feedback for natural head positioning |
| JP5208810B2 (en) | 2009-02-27 | 2013-06-12 | 株式会社東芝 | Information processing apparatus, information processing method, information processing program, and network conference system |
| US8477175B2 (en) * | 2009-03-09 | 2013-07-02 | Cisco Technology, Inc. | System and method for providing three dimensional imaging in a network environment |
| US8659637B2 (en) | 2009-03-09 | 2014-02-25 | Cisco Technology, Inc. | System and method for providing three dimensional video conferencing in a network environment |
| US11726332B2 (en) | 2009-04-27 | 2023-08-15 | Digilens Inc. | Diffractive projection apparatus |
| US9335604B2 (en) | 2013-12-11 | 2016-05-10 | Milan Momcilo Popovich | Holographic waveguide display |
| US8659639B2 (en) | 2009-05-29 | 2014-02-25 | Cisco Technology, Inc. | System and method for extending communications between participants in a conferencing environment |
| US10884607B1 (en) | 2009-05-29 | 2021-01-05 | Steelcase Inc. | Personal control apparatus and method for sharing information in a collaborative workspace |
| US8633999B2 (en) * | 2009-05-29 | 2014-01-21 | DigitalOptics Corporation Europe Limited | Methods and apparatuses for foreground, top-of-the-head separation from background |
| CZ305294B6 (en) * | 2009-06-03 | 2015-07-22 | Masarykova Univerzita | Video conference environment for communication of remote groups and communication method of remote group in such video conference environment |
| US9082297B2 (en) | 2009-08-11 | 2015-07-14 | Cisco Technology, Inc. | System and method for verifying parameters in an audiovisual environment |
| US11300795B1 (en) | 2009-09-30 | 2022-04-12 | Digilens Inc. | Systems for and methods of using fold gratings coordinated with output couplers for dual axis expansion |
| US11320571B2 (en) | 2012-11-16 | 2022-05-03 | Rockwell Collins, Inc. | Transparent waveguide display providing upper and lower fields of view with uniform light extraction |
| US8233204B1 (en) | 2009-09-30 | 2012-07-31 | Rockwell Collins, Inc. | Optical displays |
| US10795160B1 (en) | 2014-09-25 | 2020-10-06 | Rockwell Collins, Inc. | Systems for and methods of using fold gratings for dual axis expansion |
| EP2317764A1 (en) | 2009-11-03 | 2011-05-04 | Deutsche Telekom AG | Method and system for presenting image data |
| US8520051B2 (en) * | 2009-12-17 | 2013-08-27 | Alcatel Lucent | Videoconferencing terminal with a persistence of vision display and a method of operation thereof to maintain eye contact |
| CN102884419B (en) * | 2010-01-26 | 2016-08-10 | 戴比尔斯英国有限公司 | Gemstone sparkle is analyzed |
| US8659826B1 (en) | 2010-02-04 | 2014-02-25 | Rockwell Collins, Inc. | Worn display system and method without requiring real time tracking for boresight precision |
| US9225916B2 (en) | 2010-03-18 | 2015-12-29 | Cisco Technology, Inc. | System and method for enhancing video images in a conferencing environment |
| USD628175S1 (en) | 2010-03-21 | 2010-11-30 | Cisco Technology, Inc. | Mounted video unit |
| USD626102S1 (en) | 2010-03-21 | 2010-10-26 | Cisco Tech Inc | Video unit with integrated features |
| USD628968S1 (en) | 2010-03-21 | 2010-12-14 | Cisco Technology, Inc. | Free-standing video unit |
| USD626103S1 (en) | 2010-03-21 | 2010-10-26 | Cisco Technology, Inc. | Video unit with integrated features |
| US9955209B2 (en) | 2010-04-14 | 2018-04-24 | Alcatel-Lucent Usa Inc. | Immersive viewer, a method of providing scenes on a display and an immersive viewing system |
| US20110270663A1 (en) * | 2010-04-30 | 2011-11-03 | American Teleconferncing Services Ltd. | Location-Aware Conferencing With Participant Rewards |
| US9294716B2 (en) | 2010-04-30 | 2016-03-22 | Alcatel Lucent | Method and system for controlling an imaging system |
| US9313452B2 (en) | 2010-05-17 | 2016-04-12 | Cisco Technology, Inc. | System and method for providing retracting optics in a video conferencing environment |
| US8971628B2 (en) | 2010-07-26 | 2015-03-03 | Fotonation Limited | Face detection using division-generated haar-like features for illumination invariance |
| US8421844B2 (en) | 2010-08-13 | 2013-04-16 | Alcatel Lucent | Apparatus for correcting gaze, a method of videoconferencing and a system therefor |
| US8896655B2 (en) | 2010-08-31 | 2014-11-25 | Cisco Technology, Inc. | System and method for providing depth adaptive video conferencing |
| US8599934B2 (en) | 2010-09-08 | 2013-12-03 | Cisco Technology, Inc. | System and method for skip coding during video conferencing in a network environment |
| US8754925B2 (en) | 2010-09-30 | 2014-06-17 | Alcatel Lucent | Audio source locator and tracker, a method of directing a camera to view an audio source and a video conferencing terminal |
| US8599865B2 (en) | 2010-10-26 | 2013-12-03 | Cisco Technology, Inc. | System and method for provisioning flows in a mobile network environment |
| CN101986346B (en) * | 2010-11-01 | 2012-08-08 | 华为终端有限公司 | Face image processing method and device |
| US8699457B2 (en) | 2010-11-03 | 2014-04-15 | Cisco Technology, Inc. | System and method for managing flows in a mobile network environment |
| US8730297B2 (en) | 2010-11-15 | 2014-05-20 | Cisco Technology, Inc. | System and method for providing camera functions in a video environment |
| US8902244B2 (en) | 2010-11-15 | 2014-12-02 | Cisco Technology, Inc. | System and method for providing enhanced graphics in a video environment |
| US9338394B2 (en) | 2010-11-15 | 2016-05-10 | Cisco Technology, Inc. | System and method for providing enhanced audio in a video environment |
| US9143725B2 (en) | 2010-11-15 | 2015-09-22 | Cisco Technology, Inc. | System and method for providing enhanced graphics in a video environment |
| US8542264B2 (en) | 2010-11-18 | 2013-09-24 | Cisco Technology, Inc. | System and method for managing optics in a video environment |
| US8723914B2 (en) | 2010-11-19 | 2014-05-13 | Cisco Technology, Inc. | System and method for providing enhanced video processing in a network environment |
| US9111138B2 (en) | 2010-11-30 | 2015-08-18 | Cisco Technology, Inc. | System and method for gesture interface control |
| USD682864S1 (en) | 2010-12-16 | 2013-05-21 | Cisco Technology, Inc. | Display screen with graphical user interface |
| USD682293S1 (en) | 2010-12-16 | 2013-05-14 | Cisco Technology, Inc. | Display screen with graphical user interface |
| USD678320S1 (en) | 2010-12-16 | 2013-03-19 | Cisco Technology, Inc. | Display screen with graphical user interface |
| USD678308S1 (en) | 2010-12-16 | 2013-03-19 | Cisco Technology, Inc. | Display screen with graphical user interface |
| USD682854S1 (en) | 2010-12-16 | 2013-05-21 | Cisco Technology, Inc. | Display screen for graphical user interface |
| USD678894S1 (en) | 2010-12-16 | 2013-03-26 | Cisco Technology, Inc. | Display screen with graphical user interface |
| USD678307S1 (en) | 2010-12-16 | 2013-03-19 | Cisco Technology, Inc. | Display screen with graphical user interface |
| USD682294S1 (en) | 2010-12-16 | 2013-05-14 | Cisco Technology, Inc. | Display screen with graphical user interface |
| US8692862B2 (en) | 2011-02-28 | 2014-04-08 | Cisco Technology, Inc. | System and method for selection of video data in a video conference environment |
| WO2012136970A1 (en) | 2011-04-07 | 2012-10-11 | Milan Momcilo Popovich | Laser despeckler based on angular diversity |
| US8670019B2 (en) | 2011-04-28 | 2014-03-11 | Cisco Technology, Inc. | System and method for providing enhanced eye gaze in a video conferencing environment |
| US8786631B1 (en) | 2011-04-30 | 2014-07-22 | Cisco Technology, Inc. | System and method for transferring transparency information in a video environment |
| US8934026B2 (en) | 2011-05-12 | 2015-01-13 | Cisco Technology, Inc. | System and method for video coding in a dynamic environment |
| WO2016020630A2 (en) | 2014-08-08 | 2016-02-11 | Milan Momcilo Popovich | Waveguide laser illuminator incorporating a despeckler |
| US10670876B2 (en) | 2011-08-24 | 2020-06-02 | Digilens Inc. | Waveguide laser illuminator incorporating a despeckler |
| WO2013027004A1 (en) | 2011-08-24 | 2013-02-28 | Milan Momcilo Popovich | Wearable data display |
| US9538133B2 (en) * | 2011-09-23 | 2017-01-03 | Jie Diao | Conveying gaze information in virtual conference |
| US8634139B1 (en) | 2011-09-30 | 2014-01-21 | Rockwell Collins, Inc. | System for and method of catadioptric collimation in a compact head up display (HUD) |
| US9366864B1 (en) | 2011-09-30 | 2016-06-14 | Rockwell Collins, Inc. | System for and method of displaying information without need for a combiner alignment detector |
| US9715067B1 (en) | 2011-09-30 | 2017-07-25 | Rockwell Collins, Inc. | Ultra-compact HUD utilizing waveguide pupil expander with surface relief gratings in high refractive index materials |
| US9507150B1 (en) | 2011-09-30 | 2016-11-29 | Rockwell Collins, Inc. | Head up display (HUD) using a bent waveguide assembly |
| US8947493B2 (en) | 2011-11-16 | 2015-02-03 | Cisco Technology, Inc. | System and method for alerting a participant in a video conference |
| US9008487B2 (en) | 2011-12-06 | 2015-04-14 | Alcatel Lucent | Spatial bookmarking |
| US8682087B2 (en) | 2011-12-19 | 2014-03-25 | Cisco Technology, Inc. | System and method for depth-guided image filtering in a video conference environment |
| EP2610855A1 (en) * | 2011-12-28 | 2013-07-03 | Samsung Electronics Co., Ltd | Display apparatus |
| WO2013102759A2 (en) | 2012-01-06 | 2013-07-11 | Milan Momcilo Popovich | Contact image sensor using switchable bragg gratings |
| EP3429195A1 (en) * | 2012-02-27 | 2019-01-16 | Perceptiko AG | Method and system for image processing in video conferencing for gaze correction |
| US9523852B1 (en) | 2012-03-28 | 2016-12-20 | Rockwell Collins, Inc. | Micro collimator system and method for a head up display (HUD) |
| US9369667B2 (en) * | 2012-04-11 | 2016-06-14 | Jie Diao | Conveying gaze information in virtual conference |
| EP2842003B1 (en) | 2012-04-25 | 2019-02-27 | Rockwell Collins, Inc. | Holographic wide angle display |
| JP2013235562A (en) * | 2012-05-04 | 2013-11-21 | Commonwealth Scientific & Industrial Research Organization | System and method for eye position on video |
| US8902281B2 (en) | 2012-06-29 | 2014-12-02 | Alcatel Lucent | System and method for image stabilization in videoconferencing |
| US20140098296A1 (en) * | 2012-10-04 | 2014-04-10 | Ati Technologies Ulc | Method and apparatus for changing a perspective of a video |
| US9933684B2 (en) * | 2012-11-16 | 2018-04-03 | Rockwell Collins, Inc. | Transparent waveguide display providing upper and lower fields of view having a specific light output aperture configuration |
| US9265458B2 (en) | 2012-12-04 | 2016-02-23 | Sync-Think, Inc. | Application of smooth pursuit cognitive testing paradigms to clinical drug development |
| US9681154B2 (en) | 2012-12-06 | 2017-06-13 | Patent Capital Group | System and method for depth-guided filtering in a video conference environment |
| KR20140090538A (en) | 2013-01-09 | 2014-07-17 | 삼성전자주식회사 | Display apparatus and controlling method thereof |
| US9380976B2 (en) | 2013-03-11 | 2016-07-05 | Sync-Think, Inc. | Optical neuroinformatics |
| US9674413B1 (en) | 2013-04-17 | 2017-06-06 | Rockwell Collins, Inc. | Vision system and method having improved performance and solar mitigation |
| WO2014178047A1 (en) * | 2013-04-30 | 2014-11-06 | Inuitive Ltd. | System and method for video conferencing |
| JP6229314B2 (en) | 2013-05-30 | 2017-11-15 | ソニー株式会社 | Information processing apparatus, display control method, and program |
| US9727772B2 (en) | 2013-07-31 | 2017-08-08 | Digilens, Inc. | Method and apparatus for contact image sensing |
| US9325936B2 (en) * | 2013-08-09 | 2016-04-26 | Samsung Electronics Co., Ltd. | Hybrid visual communication |
| US9244281B1 (en) | 2013-09-26 | 2016-01-26 | Rockwell Collins, Inc. | Display system and method using a detached combiner |
| US10732407B1 (en) | 2014-01-10 | 2020-08-04 | Rockwell Collins, Inc. | Near eye head up display system and method with fixed combiner |
| US9519089B1 (en) | 2014-01-30 | 2016-12-13 | Rockwell Collins, Inc. | High performance volume phase gratings |
| US9244280B1 (en) | 2014-03-25 | 2016-01-26 | Rockwell Collins, Inc. | Near eye display system and method for display enhancement or redundancy |
| US9843713B2 (en) * | 2014-04-02 | 2017-12-12 | Nebulys Technologies, Inc. | Systems and methods for video communication |
| WO2016020632A1 (en) | 2014-08-08 | 2016-02-11 | Milan Momcilo Popovich | Method for holographic mastering and replication |
| US10241330B2 (en) | 2014-09-19 | 2019-03-26 | Digilens, Inc. | Method and apparatus for generating input images for holographic waveguide displays |
| TW201615009A (en) * | 2014-09-24 | 2016-04-16 | Thomson Licensing | A background light enhancing apparatus responsive to a local camera output video signal |
| WO2016045924A1 (en) * | 2014-09-24 | 2016-03-31 | Thomson Licensing | A background light enhancing apparatus responsive to a remotely generated video signal |
| US9715110B1 (en) | 2014-09-25 | 2017-07-25 | Rockwell Collins, Inc. | Automotive head up display (HUD) |
| US10088675B1 (en) | 2015-05-18 | 2018-10-02 | Rockwell Collins, Inc. | Turning light pipe for a pupil expansion system and method |
| EP3245444B1 (en) | 2015-01-12 | 2021-09-08 | DigiLens Inc. | Environmentally isolated waveguide display |
| US9541998B2 (en) | 2015-01-29 | 2017-01-10 | Samsung Electronics Co., Ltd. | Electronic system with gaze alignment mechanism and method of operation thereof |
| US9632226B2 (en) | 2015-02-12 | 2017-04-25 | Digilens Inc. | Waveguide grating device |
| US10247943B1 (en) | 2015-05-18 | 2019-04-02 | Rockwell Collins, Inc. | Head up display (HUD) using a light pipe |
| US11366316B2 (en) | 2015-05-18 | 2022-06-21 | Rockwell Collins, Inc. | Head up display (HUD) using a light pipe |
| US10126552B2 (en) | 2015-05-18 | 2018-11-13 | Rockwell Collins, Inc. | Micro collimator system and method for a head up display (HUD) |
| KR102317021B1 (en) * | 2015-06-26 | 2021-10-25 | 삼성전자주식회사 | Display apparatus and image correction method thereof |
| US10108010B2 (en) | 2015-06-29 | 2018-10-23 | Rockwell Collins, Inc. | System for and method of integrating head up displays and head down displays |
| US10690916B2 (en) | 2015-10-05 | 2020-06-23 | Digilens Inc. | Apparatus for providing waveguide displays with two-dimensional pupil expansion |
| US9743040B1 (en) * | 2015-12-03 | 2017-08-22 | Symantec Corporation | Systems and methods for facilitating eye contact during video conferences |
| US10598932B1 (en) | 2016-01-06 | 2020-03-24 | Rockwell Collins, Inc. | Head up display for integrating views of conformally mapped symbols and a fixed image source |
| JP6895451B2 (en) | 2016-03-24 | 2021-06-30 | ディジレンズ インコーポレイテッド | Methods and Devices for Providing Polarized Selective Holography Waveguide Devices |
| JP6734933B2 (en) | 2016-04-11 | 2020-08-05 | ディジレンズ インコーポレイテッド | Holographic Waveguide Device for Structured Light Projection |
| WO2018102834A2 (en) | 2016-12-02 | 2018-06-07 | Digilens, Inc. | Waveguide device with uniform output illumination |
| US10264213B1 (en) | 2016-12-15 | 2019-04-16 | Steelcase Inc. | Content amplification system and method |
| WO2018129398A1 (en) | 2017-01-05 | 2018-07-12 | Digilens, Inc. | Wearable heads up displays |
| US10295824B2 (en) | 2017-01-26 | 2019-05-21 | Rockwell Collins, Inc. | Head up display with an angled light pipe |
| US10560661B2 (en) | 2017-03-16 | 2020-02-11 | Dolby Laboratories Licensing Corporation | Detecting and mitigating audio-visual incongruence |
| JP7399084B2 (en) | 2017-10-16 | 2023-12-15 | ディジレンズ インコーポレイテッド | System and method for doubling the image resolution of pixelated displays |
| US10656706B2 (en) * | 2017-12-04 | 2020-05-19 | International Business Machines Corporation | Modifying a computer-based interaction based on eye gaze |
| US10914950B2 (en) | 2018-01-08 | 2021-02-09 | Digilens Inc. | Waveguide architectures and related methods of manufacturing |
| CN111566571B (en) | 2018-01-08 | 2022-05-13 | 迪吉伦斯公司 | System and method for holographic grating high throughput recording in waveguide cells |
| WO2020023779A1 (en) | 2018-07-25 | 2020-01-30 | Digilens Inc. | Systems and methods for fabricating a multilayer optical structure |
| WO2020089971A1 (en) * | 2018-10-29 | 2020-05-07 | 有限会社アドリブ | Image processing apparatus, method, and computer program |
| US11022794B2 (en) * | 2018-12-27 | 2021-06-01 | Facebook Technologies, Llc | Visual indicators of user attention in AR/VR environment |
| US11112865B1 (en) * | 2019-02-13 | 2021-09-07 | Facebook Technologies, Llc | Systems and methods for using a display as an illumination source for eye tracking |
| JP2022520472A (en) | 2019-02-15 | 2022-03-30 | ディジレンズ インコーポレイテッド | Methods and equipment for providing holographic waveguide displays using integrated grids |
| US20200292745A1 (en) | 2019-03-12 | 2020-09-17 | Digilens Inc. | Holographic Waveguide Backlight and Related Methods of Manufacturing |
| JP2022535460A (en) | 2019-06-07 | 2022-08-08 | ディジレンズ インコーポレイテッド | Waveguides incorporating transmission and reflection gratings, and associated fabrication methods |
| JP2022543571A (en) | 2019-07-29 | 2022-10-13 | ディジレンズ インコーポレイテッド | Method and Apparatus for Multiplying Image Resolution and Field of View for Pixelated Displays |
| US11442222B2 (en) | 2019-08-29 | 2022-09-13 | Digilens Inc. | Evacuated gratings and methods of manufacturing |
| EP4028853A4 (en) * | 2019-09-10 | 2023-06-07 | INTEL Corporation | Laptop computers with movable accessory housing |
| JP7420585B2 (en) | 2020-02-20 | 2024-01-23 | 日本放送協会 | AR display control device, its program, and AR display system |
| US11783531B2 (en) * | 2020-12-01 | 2023-10-10 | Matsuko S.R.O. | Method, system, and medium for 3D or 2.5D electronic communication |
| US20230177879A1 (en) * | 2021-12-06 | 2023-06-08 | Hewlett-Packard Development Company, L.P. | Videoconference iris position adjustments |
| DE102022121634A1 (en) * | 2022-08-26 | 2024-02-29 | Telefónica Germany GmbH & Co. OHG | System, method, computer program and computer-readable medium |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2312904A1 (en) * | 1975-05-27 | 1976-12-24 | Poirier Alain | IMPROVEMENTS TO TELECONFERENCE SYSTEMS AND DEVICES VIA VIDEO PHONE |
| US5231674A (en) * | 1989-06-09 | 1993-07-27 | Lc Technologies, Inc. | Eye tracking method and apparatus |
| DE4102895C1 (en) * | 1991-01-31 | 1992-01-30 | Siemens Ag, 8000 Muenchen, De | |
| CA2081041C (en) * | 1992-01-03 | 1996-08-06 | Robert Louis Beecher | Camera field of view indicator |
| JPH05300499A (en) * | 1992-04-17 | 1993-11-12 | Toshiba Corp | Visual telephone system |
| GB2273411A (en) * | 1992-12-11 | 1994-06-15 | Ibm | Video conferencing terminal |
| US5359362A (en) * | 1993-03-30 | 1994-10-25 | Nec Usa, Inc. | Videoconference system using a virtual camera image |
| US5400069A (en) * | 1993-06-16 | 1995-03-21 | Bell Communications Research, Inc. | Eye contact video-conferencing system and screen |
| US5438357A (en) * | 1993-11-23 | 1995-08-01 | Mcnelley; Steve H. | Image manipulating teleconferencing system |
-
1994
- 1994-10-25 US US08/328,640 patent/US5500671A/en not_active Expired - Lifetime
-
1995
- 1995-09-06 CA CA002157613A patent/CA2157613C/en not_active Expired - Fee Related
- 1995-09-11 TW TW084109459A patent/TW297985B/zh active
- 1995-10-18 GB GB9521385A patent/GB2294605B/en not_active Expired - Fee Related
- 1995-10-20 DE DE19539048A patent/DE19539048B4/en not_active Expired - Fee Related
- 1995-10-25 JP JP7299286A patent/JPH08237629A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| TW297985B (en) | 1997-02-11 |
| JPH08237629A (en) | 1996-09-13 |
| GB2294605A (en) | 1996-05-01 |
| GB9521385D0 (en) | 1995-12-20 |
| DE19539048A1 (en) | 1996-05-02 |
| CA2157613A1 (en) | 1996-04-26 |
| DE19539048B4 (en) | 2006-04-06 |
| GB2294605B (en) | 1997-01-15 |
| US5500671A (en) | 1996-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2157613C (en) | Video conference system and method of providing parallax correction and a sense of presence | |
| US8299979B2 (en) | Integral eye-path alignment on telephony and computer video devices using two or more image sensing devices | |
| EP3510768B1 (en) | Three-dimensional telepresence system | |
| US5438357A (en) | Image manipulating teleconferencing system | |
| US6583808B2 (en) | Method and system for stereo videoconferencing | |
| Zhang et al. | Viewport: A distributed, immersive teleconferencing system with infrared dot pattern | |
| US6417880B1 (en) | Stereoscopic CG image generating apparatus and stereoscopic TV apparatus | |
| US6275258B1 (en) | Voice responsive image tracking system | |
| WO2001010121A1 (en) | Method and apparatus for enabling a videoconferencing participant to appear focused on camera to corresponding users | |
| US20050083248A1 (en) | Mobile face capture and image processing system and method | |
| KR102615844B1 (en) | A portable system that allows blind or visually impaired people to understand their surroundings through sound or touch. | |
| KR100307854B1 (en) | Method and Apparatus of Gaze Compensation for Eye Contact Using Single Camera | |
| US6836286B1 (en) | Method and apparatus for producing images in a virtual space, and image pickup system for use therein | |
| US20170237941A1 (en) | Realistic viewing and interaction with remote objects or persons during telepresence videoconferencing | |
| KR20130138533A (en) | 3d video-teleconferencing apparatus capable of eye contact and method using the same | |
| US6906742B1 (en) | Maintaining eye-contact in teleconferencing using structured light | |
| Breiteneder et al. | TELEPORT—an augmented reality teleconferencing environment | |
| JP3464754B2 (en) | Method and apparatus for synthesizing a face image of a person wearing a head-mounted display | |
| WO2017141584A1 (en) | Information processing apparatus, information processing system, information processing method, and program | |
| JP4501037B2 (en) | COMMUNICATION CONTROL SYSTEM, COMMUNICATION DEVICE, AND COMMUNICATION METHOD | |
| JPH08256316A (en) | Communication conference system | |
| JPH11355804A (en) | Network conference image processing unit | |
| CN113632458A (en) | System, algorithm and design for wide angle camera perspective experience | |
| JP2556925B2 (en) | Three-dimensional conversation communication device | |
| CN116954349A (en) | MR element universe remote psychological consultation AI interactive immersion system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKLA | Lapsed |