WO2001030079A1

WO2001030079A1 - Camera with peripheral vision

Info

Publication number: WO2001030079A1
Application number: PCT/EP2000/009770
Authority: WO
Inventors: Hugo Strubbe
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 1999-10-19
Filing date: 2000-10-04
Publication date: 2001-04-26
Also published as: JP2003512783A; EP1142332A1

Abstract

A camera viewing system for direct and peripheral vision is provided comprising at least one dual opposing camera assembly mounted in back-to-back relationship, preferably on the same optical axis, at least one camera assembly providing at least a hemispherical field of view and at least one camera assembly providing a direct forward field of view that is less than hemispherical.

Description

Camera with peripheral vision

FIELD OF THE INVENTION

This invention relates to a camera viewing system with peripheral vision.

BACKGROUND OF THE INVENTION Camera viewing systems are now used for a variety of purposes including surveillance, security, video conferencing, etc., and, ideally, allow for simultaneous remote viewing, close viewing, and wide-angle and include systems in which a camera converts the optical image into an electrical signal, the image signals are digitized electronically, the digitized images are transformed by image processors controlled by a microcomputer into two or three dimensional images which are stored in an output image buffer or system, and the stored images are in turn scanned out by display electronics to a video display device for viewing. One such system is described in WO 97/43854 wherein the image of a substantially hemispherical scene is sensed using a truncated, convex, substantially paraboloid-shaped reflector positioned to orthographically reflect an image of a hemispherical scene. There are drawbacks associated with all of such systems. For example, conventional imaging systems which include a camera with a lens that provides a perspective view of an image, even with a very wide angle lens, have a limited field of view, (i.e. covering less than a full hemisphere). This limited field of view may be expanded by tilting and panning as is well known in the art but such a system suffers from drawbacks including the significant amount of time required to make a full rotation to view the expanded view making the system unsuitable for real-time applications. Omni-directional cameras can capture a hemispherical field of view but the captured images have limited resolution and a blind spot on top where the camera is mounted. Systems which employ multiple cameras with a corresponding reflector or with multiple reflectors pointing in different directions, may also be employed to achieve a large field of view. In practice, however, it is difficult to design such a configuration in which the centers of projection of all the imaging systems coincide so that a true omnidirectional system may be achieved. A single planar mirror does not enhance the field of view of a traditional imaging system. Multiple planar mirrors (each pointing in a different direction) can be used with a single imaging system. However, for any 2 given imaging system, each planar mirror produces a different center of projection. Multiple planar mirrors therefore require the use of multiple imaging systems (multiple lens and multiple image detectors). A fish-eye lens can also be used to obtain a hemispherical field of view. Such fish-eye lens cameras have a short focal length and can expand the field of view to be as large as a hemisphere but are larger and more complex than conventional lenses and often distort the center field of view (as well as the periphery). However, commercial fish- eye lenses do not guarantee a single center of projection, i.e. scene points are imaged from slightly different directions. Moreover, it is difficult if not impossible to obtain distortion-free images from the acquired wide-angle image. Due to their complex structures, fish-eye lenses also tend to be significantly more expensive than conventional lenses or mirrors.

Additionally, systems using multiple planar mirrors arranged in the shape of a pyramid in conjunction with CCD cameras viewing more than 90 degrees by 50 degrees of the hemispherical scene suffer from the disadvantage of requiring multiple sensors to capture a hemispherical image and distortion at the "seams" when the separable views are combined to provide a full 360 degree view. Curved reflective surfaces generate multiple viewpoints. Systems in which conventional cameras are closely mounted beside a camera with a fish-eye lens introduces a parallax between them; and systems that use internal optics to provide wide viewing angles and that rely on the movement of either a mirror or prism to change the tilt- angle of orientation to minimize distortion often generate blind spots in the center of the view. Additionally, when seeking improvements in the existing systems, many of the systems discussed above seek to enhance the peripheral view and do not provide for a close-up view. With teleconferencing in particular, where some type of display is exhibited at one location, and persons at a distant location desire to view all or a selected portion of the display in a close-up view, it is desirable to have a viewing system that can focus on one or more participants in a close-up view but still be aware of what is going on in the peripheral area around the participants. A camera viewing system with peripheral vision and a close-up view, analogous to human vision, is needed.

There is a need in the industry for camera viewing systems that efficiently capture and display visual information within a hemispheric or panoramic field of view containing particularly peripheral content, and that permit electronic manipulation and display of the image while minimizing distortion effects and other disadvantages of the prior art described above. SUMMARY OF THE INVENTION

An object of the invention is to provide camera viewing systems with a panoramic or hemispherical field of view with peripheral vision that efficiently capture and display visual information within a substantially hemispheric or entire field of view containing particularly peripheral content, and that permit electronic manipulation and display of the image while minimizing distortion effects.

These and other objects of the invention are accomplished, according to a first embodiment of the invention, wherein a computer vision system for producing a substantially hemispheric or panoramic image is provided comprising a camera viewing system adapted to provide a substantially panoramic or hemispherical image wherein the camera viewing system comprises at least one dual opposing camera assembly in which at least two cameras or camera assemblies are mounted in back-to-back relationship, i.e. on the same optical axis, at least one said camera or camera assembly providing at least a hemispherical field of view and at least one said camera or camera assembly providing a direct forward field of view that is less than hemispherical to collectively display a substantially hemispherical or entire field of view, and provide at least one image signal representative of said image; and further comprising an image signal processing apparatus coupled to receive said at least one image signal to convert said image signal into image signal data; means to convert said image signal data to form a video image, either analog or digital; and a display device to permit the viewing of the video data.

Preferably, a camera viewing system is provided which comprises: at least one dual opposing camera assembly having a first camera effective to provide a direct forward field of view that is less than hemispherical, and a second camera effective to provide a hemispherical field of view that includes areas of a field of view that is peripheral to the field of view of said first camera, said first and second cameras being mounted in back- to-back relationship one to another.

In a preferred embodiment, a computer vision system is provided which comprises a camera viewing system in which a first camera or camera and lens assembly is effective to provide a direct forward field of view that is less than hemispherical, a second camera or camera and lens assembly which has the same optical axis as the first camera or camera and lens assembly is arranged in back-to-back relationship therewith; a mirror is arranged to reflect light from a hemispherical field of view, said second camera or camera and lens assembly and said mirror being effective to provide a reflected hemispherical field of view that includes areas of a field of view that is peripheral to the field of view of said first camera or camera and lens assembly. The second camera or camera and lens assembly is arranged to receive light that is reflected from the mirror, i.e. the second camera or camera and lens assembly "looks off the mirror. The mirror is oriented to a line which is the common optical axis of the back-to-back mounted first and second camera or camera and lens assemblies.

Cameras and camera and lens assemblies (hereafter collectively referred to as camera assembly or camera assemblies) suitable for use in this invention may be any optical device that is capable of receiving a focussed image from any source (preferably from a lens) and transforming that image into an electronic signal or into hard copy storage such as photographic film. Any video source and image capturing device that captures an image and converts it into memory, electronic or otherwise, can serve as the first camera assembly herein. The image can be stored onto an assortment of media, e.g., photographic film, electronic storage, or other conventional storage means. Electronic storage is preferred because of the ease of electronic manipulation thereof and the invention is further described herein in terms of electronic media. However, it will be understood that the invention is equally applicable to photographic film, in which case the image processing apparatus will include an image scanner or other capture-and-conversion method to change the image into electronic format so that electronic manipulation may be performed. The system may also include means for digitizing an incoming video image signal, transforming a portion of the video image, if necessary, and producing one or more output images for viewing in a teleconferencing environment. Input and output buffers can be constructed using commercially available video random access memory chips. A control interface can be accomplished using microcontrollers and transform processors, image filtering and display by means well known in the art. Transmission may be via telephone lines, with the system including signal compression and decompression units, or wireless transmission with appropriate broadcasting apparatus may be utilized if desired.

The display device may be a monitor, television, virtual reality helmet or other virtual reality display system or a projection display. Preferably, the display device is a monitor, which may be conventional or a specially designed modular LCD monitor; it may be a single-view or multi-view monitor; and most preferably is a split screen monitor. The monitor may also comprise a picture-in-picture feature, if desired.

The split-screen monitor is particularly preferred when used with the imaging systems of the invention since such a monitor enables the user to zoom in on the speaker ("X" in the figures) while retaining the peripheral scene ("y" or "yy" and "z" or "zz" in the figures) and while maintaining a high resolution of both areas of the substantially panoraic or hemispheric scene in real-time.

Each camera assembly is schematically illustrated as comprising a lens and an imaging device such as a CCD imager. A conventional camera with zoom feature has been used with particular success as the first camera assembly. Additionally, various types of cameras for wide-angle viewing such as NTSC, and HDTV-type cameras may be used. The camera assembly can be mounted and supported in conventional manner.

The second camera assembly will be any device that captures or senses an image or receives light within a hemispherical field of view and may be a conventional camcorder or an omnidirectional video camera which provides a digital video signal output. Preferably, each sensing unit, will be an assembly commonly referred to as an omnicam and comprised of an off-the-shelf video camera, an off-the-shelf lens, and a curved mirror which provides a single effective center of projection for the complete sensing unit. This allows the construction of distortion-free images for any user selected portion of the acquired omnidirectional image.

The mirror may be any suitable reflector of any suitable configuration that is capable of receiving light from a hemispherical field of view and reflecting the same to be received by the second camera assembly. Suitable examples of mirrors and of suitable shapes therefor are preferably a substantially paraboloid-shaped reflector as described in WO 97/43854 positioned to orthographically reflect an image of a substantially hemispheric scene. Other mirrors or reflectors suitable for use include a conical reflecting surface and/or hyperboloidal mirrors. The omnicam sensing unit is capable of imaging a complete hemispherical field of view. The curved mirror used to accomplish this may be hyperbolic when the imaging projection used is perspective, and parabolic when the imaging projection used is orthographic. Preferably, the omnicam will include a telecentric lens to achieve orthographic projection and a parabolic mirror to achieve a hemispherical field of view. This configuration offers calibration advantages over one that uses a hyperbolic mirror.

Preferably, the second camera assembly is an omni-directional camera and lens assembly commonly designated an omnicam, which covers nearly a half sphere as the peripheral viewing device, and said first camera assembly is a camera with pan, tilt and zoom features or a CCD camera and lens assembly mounted in such a manner that it picks up the area that the omnicam cannot see, and arranged in back-to-back relationship on the same axis whereby the first camera assembly looks forward, and the second camera assembly looks at the hemisphere. In yet another preferred embodiment, a standard camera is mounted in back- to-back relationship with an omnicam camera and related components to produce a computer vision system that has an undistorted center view and a peripheral area view. Such a system may focus on one object or subject, while providing image data on peripheral objects or subjects. The undistorted center view and the peripheral area view are aligned such that parallax error between the two views is minimized and preferably is eliminated. The camera assemblies are preferably mounted on a pan, and tilt "PT" drive mechanism.

The camera viewing system is eminently suitable for use in video conferencing and security systems and is substantially free of the disadvantage of prior systems discussed above.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a side view of an exemplary embodiment of an omni-directional imaging apparatus of this invention; Figure 2 is a perspective view of a split-screen monitor according to one embodiment of the invention;

Figure 3 is a perspective view of a split-screen monitor according to a second embodiment of the invention; and

Figure 4 is a block diagram of a computer vision system according to an embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention will be better understood with reference to the details of specific embodiments that follow.

With reference to Fig.l, there is illustrated a computer vision system (100) according to an exemplary embodiment of the invention in which a camera viewing system (110) has a first camera assembly comprising a camera (111) with a lens (108), the camera assembly being effective to provide a direct forward field of view that is less than hemispherical and that provides an image that at least encompasses object x, and a second camera assembly comprising a camera (112) with a lens (109) and mirror (113), the camera assembly and mirror being effective to provide a hemispherical field of view that includes areas of a field of view that is hemispherical and provides an image that includes at least objects y or yy and z or zz in a field of view that is peripheral to the field of view of said first camera assembly (111,108), said first and second camera assemblies (108,109,111,112) being mounted in back-to-back relationship one to another. The second camera assembly (112,109) receives the reflected image of the hemispherical scene from the mirror (113), which is mounted to reflect a substantially hemispherical scene. In the embodiment illustrated, the cameras (111) and (112) generate a signal which may be sent through a cable or telephone line (120) to a computer (125), all as is well known in the art. The stored electronic image data may then be accessed by a transform processor and can be electronically manipulated by panning, up/down, zooming, etc.

The second camera assembly and mirror are preferably mounted on a suitable support (114) which may be a thin wire or transparent column or tubing effective to support the camera without interfering with the image. Additionally, if desired, the entire assembly may be mounted on a PT drive mechanism.

In the computer vision system (100) illustrated in the drawing, a first camera (111), and lens (108), preferably with a zoom feature for magnification, is mounted on a pan- tilt base mechanism (116) and is effective to provide a direct forward and close-up field of view that looks forward at object X, and an omnicam second camera (112) and lens (109) looks at the hemisphere and is effective to receive from a mirror (113) a reflected substantially panoramic or hemispherical field of view that includes objects x, yy, and zz. The cameras (111,112), lenses (108,109), and mirror (113) are mounted, on a suitable camera support base (114), if desired, in back-to-back relationship along a common optical axis (115). The cameras (111,112) generate a signal which is sent through a cable or telephone line (120) to computer (125), to provide a close-up image of the selected participant or participants and a field of view that includes the peripheral environment that exists at the time of imaging. The computer 125 is programmed to allow the user to view any desired portion of the hemispherical scene, to zoom in on a selected portion, or to pan the scene in any desired manner. In the normal teleconferencing scenario using conventional computer vision systems, it is necessary and customary to zoom in to the person speaking (x) while excluding the neighboring participant (y or z) or participants (yy,zz) adjacent to the speaker from the field of vision. As a result, the context of the other persons participating in the conference is lost. Using the computer vision system of the invention for teleconferencing, it is possible to zoom in to a person that is speaking (x) without losing the context of the other participants (y,yy,z and/or zz as the case may be) or of what is happening in the area peripheral to the speaker via a high-resolution, high quality, seamless image as illustrated in the monitors 125-131 in Figs.l to 3. As is known in the art, (see U.S. Patent 5,508,734, for example), imperfections in the image representation of any field inherently result from the nature of creating an image with any spherical medium. The magnitude of these imperfections increases proportionally to the distance a point in the field is from the axis perpendicular to the optical imaging system. As the angle between the optical axis and a point in the field increases, aberrations of the corresponding image increase proportional to this angle cubed. Thus aberrations are more highly exaggerated in the peripheral areas with respect to the central areas of a hemispherical image. Conventionally, the valuable content from the peripheral area lacks image quality (resolution) and attempts to correct the same have resulted in reducing the image quality of both the central and peripheral areas.

In a particularly preferred embodiment of the invention as illustrated in Fig. 2, a split-screen monitor 130 may display images of central and peripheral areas of the panoramic or hemispherical scene of high resolution. As illustrated, the view of x (the speaker), which is data passed from the first camera assembly with zoom lens is shown in the upper % to ^l/ι of the screen, and the peripheral data content of the scene, yy and zz (other members of the teleconference or background) is shown in the lower VΛ to V-i of the screen.

Figure 4 illustrates a block diagram of one embodiment of a system that incorporates one embodiment of an image signal processing apparatus for controlling data produced by the cameras of a computer viewing system described in Figs. 1 to 3 and is similar to a control system illustrated in EPO 740,177 A2. As illustrated herein, cameras (111, 112) obtain views of an area as described herein above. The image signal or output signal of the cameras is passed through analog to digital converters (131, 132), if necessary. The output of the cameras can be thought of as a stream of pixels and the output from the converters can be thought of as data representative of the pixels from the cameras. The output of the converters is passed through MUX (134) which allows the pixel data from each converter to reach the memory (135). Controller (136) cycles the lines of MUX so that all outputs of the A D converters are stored in the memory (135). It is also possible to eliminate MUX and store the output of each A/D converter in a separate memory. It is also possible to eliminate the A/D converters when using digital cameras or sensors. Controller (136) is implemented using a microprocessor which provides control signals to counters that control the switching of MUX and counters used to provide addressing to memory (135). As the pixel information is read from memory (135), all of the information is passed over telecommunications network (137) to telecommunications bridge (138). The information from memory is provided to bridge via modem (139); however, the data may be passed to bridge (138) without use of modem (139) if a digital connection is made between the memory and the bridge. The bridge then distributes all of the data received from memory to each user in communication with the bridge. If the bridge provides analog link to users, a modem should be used at each user port. If the bridge provides digital link with each user a modem is not required. The pixel data is passed to video memory (140) which then passes the data to display (126-131) under control of user input device (125), which is preferably a computer keyboard although a mouse or joystick may also be employed.

It will be understood that the invention may be embodied in other specific forms without departing from the spirit and scope or essential characteristics thereof, the present disclosed examples being only preferred embodiments thereof.

Claims

CLAIMS:

1. A camera viewing system (110) for direct and peripheral vision comprising dual opposing camera assemblies (108,109,111,112) mounted in back-to-back relationship, at least one said camera assembly(109,112) providing at least a hemispherical field of view and at least one camera assembly (108,11 l)provi ding a direct forward field of view that is less than hemispherical.

2. A camera viewing system as claimed in Claim 1, wherein said dual opposing camera assemblies are mounted on approximately the same optical axis (115).

3. A camera viewing system (110) for direct and peripheral vision which comprises at least one dual opposing camera assembly having a first camera assembly (108,111) effective to provide a direct forward field of view that is less than hemispherical, and a second camera assembly (109,112) effective to provide a hemispherical field of view that includes areas of a field of view (Y,Z) that is peripheral to the field of view (X) of said first camera assembly, said first and second camera assemblies being mounted in back-to- back relationship one to another.

4. A camera viewing system as claimed in Claim 3, wherein said dual opposing camera assemblies are mounted on approximately the same optical axis (115).

5. A camera viewing system as claimed in Claim 3, wherein said first camera assembly comprises a camera and lens assembly (108,111) effective to provide a direct forward field of view (X) that is less than hemispherical; said second camera assembly comprises a camera and lens assembly (109,112) which lies on the same horizontal plane and has the same optical axis (115) as the first camera and lens assembly, is arranged in back-to- back relationship therewith, and further comprises a mirror (113) arranged to reflect light from a hemispherical field of view, said second camera and lens assembly and said mirror being effective to provide a reflected hemispherical field of view that includes areas of a field of view (Y,Z) that is peripheral to the field of view (X) of said first camera and lens assembly.

6. A computer vision system(lOO) which comprises a camera viewing system (110) for direct and peripheral vision which comprises at least one dual opposing camera assembly having a first camera assembly (108,111) effective to provide a direct forward field of view (X) that is less than hemispherical, and a second camera assembly (109,112) effective to provide a hemispherical field of view that includes areas of a field of view (Y,Z) that is peripheral to the field of view (X) of said first camera, said first and second camera assemblies being mounted in back-to-back relationship one to another.

7. A computer vision system as claimed in Claim 6, wherein said dual opposing camera assemblies are mounted on approximately the same optical axis (115).

8. A computer vision system a claimed in Claim 7, wherein said first camera and lens assembly (108,111) is effective to provide a direct forward field of view that is less than hemispherical, said second camera and lens assembly (109,112) lies on the same horizontal plane and has the same optical axis (115) as the first camera and lens assembly, is arranged in back-to-back relationship therewith, and comprises a mirror (113) arranged to reflect light from a hemispherical field of view, said second camera and lens assembly and said mirror being effective to provide a reflected hemispherical field of view that includes areas of a field of view that is peripheral to the field of view of said first camera and lens assembly.

9. A video conferencing system which includes a computer vision system (100) having a camera viewing system (110) for direct and peripheral vision which comprises at least one dual opposing camera assembly having a first camera assembly (108,111) effective to provide a direct forward field of view that is less than hemispherical, and a second camera assembly (109,112) effective to provide a hemispherical field of view that includes areas of a field of view that is peripheral to the field of view of said first camera assembly, said first and second camera assemblies being mounted in back-to-back relationship one to another.

10. A video conferencing system as claimed in Claim 9, wherein said dual opposing camera assemblies are mounted on approximately the same optical axis (115).

11. A video conferencing system a claimed in Claim 10, wherein said dual opposing camera assembly comprises a first camera and lens assembly (108,111) effective to provide a direct forward field of view that is less than hemispherical, said second camera assembly comprises a camera and lens assembly (109,112) which lies on the same horizontal plane and has the same optical axis (115) as the first camera and lens assembly, is arranged in back-to-back relationship therewith, and comprises a mirror (113) arranged to reflect light from a hemispherical field of view, said second camera and lens assembly and said mirror being effective to provide a reflected hemispherical field of view that includes areas of a field of view (Y,Z) that is peripheral to the field of view (X) of said first camera and lens assembly.