WO2000079779A2

WO2000079779A2 - System and method for differentiating between image foreground and background in an image based on an invisible keying signal

Info

Publication number: WO2000079779A2
Application number: PCT/IB2000/000896
Authority: WO
Inventors: Moshe Ben-Ezra; Shmuel Peleg
Original assignee: Yissum Research Development Company Of The Hebrew University Of Jerusalem
Priority date: 1999-06-21
Filing date: 2000-06-21
Publication date: 2000-12-28
Also published as: AU5419600A; WO2000079779A3

Abstract

An image recording studio systems (10) for differentiating between image foreground and background in an image based on an invisible keying signal that according with different polarization characteristics. A generated matte (36) is used in separating the foreground from the background which allows for substituting a background from a different image with the foreground in a composite image. A matte geneation system (30) generates a matte from images recorded by the image recording studio (10). The recorded image divided into light and dark images (33, 34), while the foreground objects (11) appearing approximately the same intensity in both images, the background in the light image (33) apears relatively light and appears to be opposite in the dark image (34). The matte generation system (30) generates a matte (36) in relation to the absolute brightness as between the light and dark images (33, 34) which may employed a non-linear sigmoid funcion.

Description

SYSTEM AND METHOD FOR DIFFERENTIATING BETWEEN IMAGE FOREGROUND AND

BACKGROUND IN AN IMAGE BASED ON AN INVISIBLE KEYING SIGNAL

FIELD OF THE INVENTION

The invention relates generally to the field of recording and generating images, and more particularly to systems and methods for differentiating between image foreground and background in an image based on an invisible keying signal. This facilitates, for example, generation of a matte that is useful in separating the foreground from the background which can allow for substituting a background from a different image with the foreground in a composite image.

BACKGROUND OF THE INVENTION

It is often desired, both in still photography and in moving pictures, to photograph one or more persons and/or objects at one point in time, separately photograph an image for use as a background, and later generate a composite image with the persons and/or objects in the foreground and the separately-photographed background image as the background. In one common technique, referred to as chroma keying, the persons and/or objects that will be included in the final image, which will be collectively referred to as "foreground," are photographed in front of a screen of predetermined color, typically blue or green to provide a foreground image. The background is photographed separately to provide a background image, and a composite image is made by removing the background color from the foreground image and substituting background image information from the background image. The color chosen for the back constitutes a visible key that facilitates differentiation between the background and the foreground in the foreground image. The objects in the foreground image that will remain in the composite image are generally referred to as "foreground," regardless of whether or not the objects will actually be in the foreground in the composite image. Generally, during chroma keying, a fuzzy separation is made in the foreground image between foreground and the portion of the foreground image that will be removed and for which background information will be substituted, which fuzzy separation is generally referred to as the matte.

A number of problems arise in connection with the chroma keying methodology. Since the in chroma keying substitution of the background image in the composite is made on the basis of color in the foreground image, the foreground image cannot contain a color that is close to that for which the background image is substituted, since, if the foreground image contains such a color, a portion of the background image will be substituted therefor in the composite. This can have two disadvantages. First, it can require that care be taken to ensure that the foreground objects not be of similar color as that used as the screen. If it is desired that a foreground object be of a color similar to that of the screen, the object may need to be provided with a different color when the foreground image is being recorded, and post processing of the composite image may be required to substitute the desired color for the original color.

A related problem, spill, is caused by light reflected off the screen onto the foreground objects when the foreground image is being recorded. This can have several undesirable effects. First, it can make matte formation difficult, since the spill can cause the matte to be somewhat indistinct. In addition, the large screen may violate the color calibration assumption of many cameras, which can affect both the brightness and color balance of the foreground image. Although, for still images, the camera that records the foreground image can be preset correctly to reduce the problem, in a moving picture in which the successive foreground images change it is difficult to do so.

A further problem arises in connection with objects that are semi-transparent, such as those made out of glass, and that would be expected to cast shadows in the composite image. Chroma keying uses intensity information to incorporate semi-transparency and shadows into the matte. In chroma keying, background color and image intensity are compared to a selected reference color from the background. Uneven background lighting conditions can cause erroneous classification of foreground objects. This problem can be alleviated to some extent in connection with still images, since an operator can select different points at which color and intensity are measured until the desired result is obtained. This might also be done for each image in a sequence of images; however, since the result may differ as between composite images, which can cause undesirable fluctuations as between successive images, which is very undesirable in movies.

SUMMARY OF THE INVENTION

The invention provides a new and improved system and method for differentiating between image foreground and background in an image based on an invisible keying signal.

In brief summary, the invention provides arrangements for differentiating between image foreground and background in an image based on an invisible keying signal, in particular, different polarization characteristics in the light representing the foreground and background. This facilitates, for example, generation of a matte that is useful in separating the foreground from the background which can allow for substituting a background from a different image with the foreground in a composite image.

Embodiments of the invention provide image recording studio systems for generating images that can be processed to provide a matte useful in separating foreground objects from a background. In one image recording studio, foreground objects are illuminated from in front (that is, towards a camera) by non-polarized light, and polarized light is provided as a back light. In another image recording studio, both foreground objects and a background screen are illuminated from in front by polarized light; for that studio, the surface of the background screen is selected to be such as, when light is reflected therefrom, to maintain the polarization characteristics of the reflected light, but surfaces of the foreground objects are selected to depolarize light reflected therefrom. In both cases, the images are recorded for, for example, processing to generate a matte useful in segregating the foreground objects from the background objects.

Embodiments of the invention further provide a matte generation system useful in generating a matte from images recorded by the image recording studio systems. In one matte generation system, an image recorded by the image recording studio system may be divided into light and dark images, either by the matte generation system itself or the image recording studio system. In the light image, the background appears relatively light, and, in the dark image, the background appears relatively dark, with the foreground objects appearing approximately the same intensity in both images. The matte generation system generates a matte in relation to the absolute brightness as between the light and dark images, in one embodiment employing a non-linear sigmoid function.

BRIEF DESCRIPTION OF THE DRAWINGS

This invention is pointed out with particularity in the appended claims. The above and further advantages of this invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

FIG. 1 schematically depicts one embodiment of an image recording studio system for recording foreground images in accordance with the invention;

FIG. 2 schematically depicts one embodiment of a matte generation system for generating a matte in connection with the invention;

FIG. 3 schematically depicts a second embodiment of an image recording studio system for recording foreground images in accordance with the invention FIGS. 4 and 5 schematically depict respective catadioptric cameras which are useful in connection with other embodiments of image recording studio systems.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

FIG. 1 schematically depicts one embodiment of an image recording studio system 10 for recording foreground images in accordance with the invention. Generally, instead of using a screen of a predetermined color as the background when the foreground image is being recorded, the system makes use of a screen that provides polarized light as the background. Generally, when polarized light is reflected off most kinds of surfaces, including skin and clothing, it loses some or all of its polarization. However, with some kinds of surfaces, such as silver-screen used for polarization based stereo, polarization is preserved. In connection with the invention, the matte is obtained by comparing two foreground images taken through different polarizing filters or a single foreground image that has been passed through a beam splitter to provide two images. The foreground objects appear the same in both images, but the polarization of the light in regions other than those occupied by the foreground objects will cause those regions to be relatively bright in one image (the "bright image"), and relatively dark in the other image (the "dark image"). The two images are aligned and the matte is generated in relation to the difference between their absolute brightness. The image of the foreground objects can be obtained from the bright image using the matte to mask off the portions other than the foreground objects. After the image of the foreground objects has been obtained, the composite image can be generated by combining that image with the background in a conventional manner. It will be appreciated that the different polarization characteristics of the light as between the foreground objects and background in the foreground provides an invisible keying signal, since the polarization characteristics are invisible to the unaided eye.

With this background, the image recording studio system 10 depicted in FIG. 1 is an arrangement for recording the foreground image. The system 30 for generating the matte from the foreground image will be described below in connection with FIG. 2. With reference to FIG. 1, in the image recording studio system 10, instead of placing the foreground object(s) 11 in front of a screen of a predetermined color, a polarized back light arrangement 12 is provided that provides polarized light of preferably relatively uniform intensity over preferably at least the entire field of view of the camera 13. The foreground object(s) 11 are illuminated by a light source 14, light from which may be unpolarized and may be of any convenient color. The polarized back light arrangement 12 includes a diffuser 15 and a polarizer screen 16, both of which preferably extend across the entire field of view of the camera 13 and one or more light sources generally identified by reference numeral 17. The diffuser 15 diffuses the light from the light source 17 across the polarizer screen 16 to preferably provide a relatively uniform illumination thereover. The polarizer screen 16, in turn, polarizes the light provided thereto by the diffuser 15 and transmits polarized light into the scene including the foreground object(s) 11 and toward the camera 13. The polarized light provided by the polarizer screen 16 may be any color, including but not limited to white, and may, but need not, be the same color as the light provided by the light source 17. In addition, it will be appreciated that several light sources 17 may be distributed behind the diffuser 15 to provide relatively uniform illumination thereover.

The camera 13, in particular the camera's lens 20 is directed toward the foreground object(s) 11 and polarized back light arrangement to facilitate recording an image thereof in a conventional manner. The image as recorded by the camera 13 comprises unpolarized light from the light source 17 as reflected off the foreground object(s) 11 as well as the polarized light as provided by the polarizer screen 16. It will be appreciated that, since the polarized light is provided as a back light, the foreground object(s) 11 will block the polarized light in the portion of the camera's field of view that is subtended by the respective foreground object(s) 11 so that, in the image as recorded by the camera 13, the polarized light is not in that region. Accordingly, the image as recorded by the camera will consist of a combination of the unpolarized light reflected off the foreground object(s) 11 as well as polarized light from the portion of the polarizer screen 16 that is not obscured by the foreground object(s) 11.

After the camera 13 has recorded an image, it is processed by a matte generation system to generate a matte that, in turn, will be used in connection with generation of a composite image that is a composite of the foreground object(s) 11 and a separately-recorded background image. FIG. 2 schematically depicts one embodiment of a matte generation system 30 for generating a matte using the image recorded by, for example, the image recording studio system 10 depicted in FIG. 1 , or any other image recording studio system that provides polarized light in regions other than those subtended by the foreground object(s) 11. With reference to FIG. 2, the matte generation system 30 includes a beam splitter 31 that splits the image 32 that had been previously recorded by the image recording studio system 10 into the light image 33 and the dark image 34. In both the light image 33 and the dark image 34 the foreground object(s) 11 appear to have approximately the same intensity, but in the light image 33 the background is light and in the dark image 34 the background is dark.

The light and dark images 33 and 34 are provided to matte generator 35, which aligns them and processes them to generate the matte 36. Since the two images have the same center of projection, the matte generator 35 can use a homography to align the two images in a conventional manner.

After the images 33 and 34 have been aligned, the matte generator 35 generates the matte. In one embodiment, the matte generator 35 makes use of a non- linear sigmoid function

1 l + e -a (x-β)

in connection with corresponding pixels of the light and dark images 33 and 34, where "x" is the absolute difference in brightness between the corresponding pixels, " " represents a slope parameter and "β" represents a center value. In one embodiment, in which the absolute difference values "x" have been normalized and are in the range from "zero" to "one," good results were obtained for α=l 5 and β=0.5. After the matte generator 35 has generated the matte 36, it can be used in connection with generating a composite image in a conventional manner.

FIG. 3 depicts another embodiment of a image recording studio system 40 suitable for recording an image for use by the matte generation system 30 described above in connection with FIG. 2. Unlike the image recording studio system 10 described above in connection with FIG. 1, which makes use of a back light arrangement 12 to provide polarized light, the image recording studio system 40 includes a lighting arrangement 41 that provides polarized light from the front, that is, from the side of the foreground obj ect(s) 42 that are generally toward the camera 43. The lighting arrangement 41 includes a light source 44, a diffuser 45 and a polarizer 46. As with the polarized back light arrangement 12, the light source 44 provides unpolarized light of a predetermined color or combination of colors to the diffuser 45. The diffuser 45 diffuses the light, preferably uniformly over the polarizer 46. The polarizer 46, in turn, polarizes the light provided thereto by the diffuser 15 and transmits polarized light into the scene and onto the foreground obj ect(s) 42. When the light is reflected off the surfaces of the foreground object(s) 42 that are to be used in the composite image for recording by the camera 43, the surfaces depolarize the light so that the light from those foreground objects as recorded by the camera 43 is not polarized. Positioned behind the foreground object(s) 42 is a screen 47 composed of a silver screen material. As with the polarizer screen 16 in the system 10 described above in connection with FIG.

1, preferably the screen 47 at least subtends the entire field of view of camera 43. Light from the lighting arrangement 41 also reflects off the screen 44, but unlike the foreground object(s) 42, when the light is reflected off the screen 47 it remains polarized. Accordingly, when the camera 43 records an image comprising the foreground object(s) 42 and the screen 47 the light from the foreground object(s) will not be polarized, whereas the light from the screen 47 Will be polarized, and so the region(s) of the image comprising the foreground object(s) will not be polarized, whereas other region(s) will be polarized, as with the image recording studio system 10

As noted above, when the polarized light provided by the lighting arrangement 41 reflects off a material such as silver screen, the light will remain polarized. One benefit of the image recording studio system 40, in which polarized light is provided from the front, instead of the back light arrangement used in system 10, is that, if the surface of a foreground object is composed of the silver screen material, after the image has been processed as describe above in connection with FIG.

2, to generate a matte, the resulting matte will be such as to cause that object to not be visible in the composite image. This can be used to provide various visual effect. For example, and with reference to FIG. 3, if the foreground object(s) comprise a base 50, a support 51 and an object 52, with the support 51 supporting the object 52 above the base 50, if the surface of support 51 is of the silver screen material, the matte that is produced will be such as to provide that the support is not visible in the composite image that is generated, so that the object 52 will appear to float above the base 50.

The invention provides a number of advantages. In particular, the invention provides an arrangement that facilitates generation of a matte that can be used in generating a composite of foreground objects from one image and a background from another image, and that avoids problems that are inherent in the chroma keying methodology that is generally used. Since the invention makes use of the different polarization properties as between light reflected off the foreground object(s) and other light provided in the foreground image, as opposed to differences in color, as is the case in connection with the chroma keying methodology, in an arrangement in accordance with the invention it would not be necessary to ensure that the colors of the foreground object(s) differ from that used as the background screen. In addition, variations in intensity do not cause problems, and generally there will be no spill to cause problems in the composite. It will be appreciated, however, that it will be desirable to provide that surfaces of foreground objects that are to be visible in the composite not be of silver screen, mirror, or similar material that does not depolarize polarized light on reflection thereof, particularly in connection with the image recording studio system 40 described above in connection with FIG. 3, since there may be some spill effect in connection therewith. The image recording studio system 10 which provides polarized light as a back light will normally not exhibit this problem. In addition, the surfaces of some materials, such as plastics, exhibit stress effects when exposed to polarized light, which can also cause undesirable artifacts particular in connection with the image recoding system 40 described above in connection with FIG. 3. Furthermore, some substances, such as minerals, rotate the polarization of polarized light, which can cause problems in connection with matte generation. Similarly, tilting the camera so as to record at image at an angle in relation to the polarization that, when processed by the matte generation system, will not allow for a distinct difference in backgrounds as between the light image 33 and dark image 34, can also cause a problem in connection with matte generation.

It will be appreciated that a number of modifications may be made to both the image recording studio system 10 and 40 and the matte generation system 30. Generally, the image recording studio system can be of any arrangement for providing that, in recording the foreground image, the foreground object(s) that are to be provided in the composite image are generally nonpolarized in the foreground image, whereas the rest of the foreground image is polarized. Although the image recording studio systems 10, 40 have been described as generating a single foreground image, and the matte generation system 30 has been described as generating therefrom light and dark images 33 and 34, it will be appreciated that the image recording studio system may instead generate light and dark images directly to be processed by a matte generator. In that case, the camera may comprise a catadioptric camera, which includes a beam splitter configured to split the incoming light and optical paths to provide separate images, which can be processed by the matte generator described above in connection with FIG. 2. FIGS. 4 and 5 schematically depict respective catadioptric cameras 50 and 60, with catadioptric camera 50 comprising a beam splitter and a single prism, and catadioptric camera 60 comprising a beam splitter and three prisms. With reference to FIG. 4, catadioptric camera 50 includes a beam splitter 51 that receives light representing the foreground image along a path represented by the arrow 52. A portion of the light received by the beam splitter 51 passes through the beam splitter 51 , as represented by arrow 53 , and is recorded by an image recording medium 54 as the light image 55. Another portion of the light reflected by the beam splitter 52, represented by arrow 56, is directed by aprism 57 onto another portion of the image recording medium 54 as the dark image 58. It will be appreciated that, since the optical path represented by arrow 56 is longer than the optical path represented by arrow 53 , there may be a slight scale difference between the light and dark images 55 and 58, which can be compensated for in a conventional manner. It will be appreciated that this scale difference can be alleviated if a separate image recording medium is provided for the light reflected from the beam splitter, so that the optical paths for the reflected and non-reflected light has the same path length.

The catadioptric camera 60, which will be described in connection with FIG. 5 , provides that optical paths for both the light and dark images will be the same, and using the same optical recording medium, thereby avoiding the scale difference in the images recorded by the camera 50 described above in connection with FIG. 4. With reference to FIG. 5, catadioptric camera 60 includes a beam splitter 61 that receives light representing the foreground image along a path represented by arrow 62. A portion of the light received by the beam splitter 61 passes through the beam splitter 61, as represented by arrow 63, and is reflected off prisms 64 and 65 before being recorded by an image recording medium 66 as the light image 67. Another portion of the light reflected by the beam splitter 61, represented by arrow 70, is directed by a prism 71 onto another portion of the image recording medium 66 as the dark image 72. In the catadioptric camera 60 depicted in FIG. 5, the optical paths represented by arrows 63 and 70 are typically the same, in which case there will be no slight scale difference between the light and dark images 67 and 72.

It will be appreciated that other modifications may also be made to both the image recording studio system 10 and 40 and the matte generation system 30. It may be desirable, for example, to utilize environment matting for dynamic objects, in which not only the shape(s) of foreground object(s) are utilized in connection with the composite image, but also reflection and refraction patterns are also utilized to provide, for example, information as to the object(s)'s environment. In that case, it may be desirable to, for example, divide the incoming light into a plurality of sets of spectral components, for example, red/green/blue or cyan/yellow/magenta and black, and generate a separate matte for each set.

In addition, although the matte generator 35 has been described as making use of a sigmoid function in connection with generation of a matte using the light and dark images 33 and 34, it will be appreciated that other convenient functions can be used. In addition, although the matte generation system 30 and catadioptric cameras 50 and 60 have been described as making use of beam splitters to divide the foreground image into light and dark images, it will be appreciated that polarizing filters may be used instead.

Furthermore, although the invention has been described as providing arrangements in which the foreground is represented by non-polarized light and background is represented by polarized light in the foreground image, it will be appreciated that the foreground may be represented by polarized light and the background by non-polarized light in the foreground image. In that case, it may be necessary to complement the matte as generated by the matte generator 35 described above in connection with FIG. 2 to provide a matte useful in separating the foreground objects from the background in the foreground image.

It will be appreciated that a system in accordance with the invention can be constructed in whole or in part from special purpose hardware or a general purpose computer system, or any combination thereof, any portion of which may be controlled by a suitable program. Any program may in whole or in part comprise part of or be stored on the system in a conventional manner, or it may in whole or in part be provided in to the system over a network or other mechanism for transferring information in a conventional manner. In addition, it will be appreciated that the system may be operated and/or otherwise controlled by means of information provided by an operator using operator input elements (not shown) which may be connected directly to the system or which may transfer the information to the system over a network or other mechanism for transferring information in a conventional manner.

The foregoing description has been limited to a specific embodiment of this invention. It will be apparent, however, that various variations and modifications may be made to the invention, with the attainment of some or all of the advantages of the invention. It is the object of the appended claims to cover these and such other variations and modifications as come within the true spirit and scope of the invention.

What is claimed as new and desired to be secured by Letters Patent is:

Claims

1. An image recording studio system for use in recording an image of at least one object in a scene comprising:

A. a light source configured to illuminate at least the at least one object;

B. background polarized light source configured to provide polarized light as at least some of a background of the at least one object, and

C. an image recording device configured to record the image of the at least one object and at least some of the background provided by the background polarized light source.

2. An image recording studio system as defined in claim 1 in which the background polarized light source comprises a polarized back light arrangement comprising at least one light source and a polarizing screen configured to project polarized light into the background.

3. An image recording studio system as defined in claim 2 in which the polarized back light arrangement further comprises a diffuser configured to diffuse light from the at least one light source over the polarizing screen.

4. An image recording studio system as defined in claim 1 in which

A. the light source is configured to illuminate the at least one object with polarized light; and

B. the background polarized light source comprises a screen whose surface maintains the polarization of the light reflected therefrom.

5. An image recording studio system as defined in claim 4 in which at least one surface of the at least one object is selected to be of a material that at least partially depolarizes light reflected therefrom.

6. An image recording studio system as defined in claim 1 in which the image recording device records a plurality of images of the at least one object, including a light image and a dark image, the background appearing relatively light in the light image and relatively dark in the dark image.

7. An image recording studio system as defined in claim 6 in which the image recording device comprises

A. an image recording medium configured to record at least one image proj ected thereupon and

B. a beam splitting arrangement configured to split light provided thereto from the at least one object into two portions for projection onto the image recording medium.

8. An image recording studio system as defined in claim 7 in which the beam splitting arrangement comprises a beam splitter.

9. An image recording studio system as defined in claim 7 in which the beam splitting arrangement includes a polarizing lens arrangement.

10. A matte generation system configured to generate a matte from a light image and a dark image, at least one object appearing substantially the same brightness in both images, with a background appearing relatively light in the light image and relatively dark in the dark image, the matte generation system including

A. light and dark image receivers configured to receive the light and dark image; and

B. a matte generator configured to generate the matte in relation to an absolute brightness as between the light and dark images.

11. A matte generation system as defined in claim 10 in which the matte generator makes use of a non-linear sigmoid function

1

1 + -a (x-β)

in connection with the light and dark images, where "x" is the absolute difference in brightness, "α" represents a slope parameter and "β" represents a center value.

12. A matte generation system as defined in claim 11 in which, for absolute difference values "x" have been normalized and are in the range from "zero" to "one," α=15 and β=0.5.

13. A matte generation system as defined in claim 10, further including a light and dark image generator for generating the light and dark images from a single image in relation to different polarization characteristics as between the at least one object and a background in said single image.

14. A matte generation system as defined in claim 13 in which the light and dark image generator comprises abeam splitting arrangement configured to split light provided thereto from the image into two portions, one portion representing the light image and the other portion representing the dark image.

15. A matt generation system as defined in claim 14 in which the beam splitting arrangement comprises a beam splitter.

16. A matt generation system as defined in claim 14 in which the beam splitting arrangement includes a polarizing lens arrangement.