WO1999000982A1

WO1999000982A1 - System for visualizing an environment

Info

Publication number: WO1999000982A1
Application number: PCT/EP1998/004000
Authority: WO
Inventors: Maurice Henk Jan Glandrup; Anne-Marie Henriette Van Soelen; Willem Van Wensveen
Original assignee: Hollandse Signaalapparaten B.V.
Priority date: 1997-06-26
Filing date: 1998-06-17
Publication date: 1999-01-07
Also published as: AU745415B2; AU8630498A; CA2295183A1; KR20010014190A; NL1006407C2; CN1261500A; JP2002511221A; EP1016283A1

Abstract

The invention relates to a system incorporating a plurality of TV cameras for producing a panoramic image of the area surrounding a ship. By means of helmet-mounted displays, the resulting image creates a virtual reality environment. The image is additionally used for point target detection and tracking.

Description

System for visualizing an environment

The invention relates to a system for visualizing an environment with the aid of TV cameras and at least one display unit. Systems of this type are well-known in the art and may for instance be used for surveillance purposes. To this end, a limited number of azimuth directions is monitored with the aid of several TV cameras. Images from these TV cameras are displayed simultaneously on several display screens; it may also occur that only one display screen is available and a selector switch for selecting a TV camera.

The drawback of prior art systems is that in this manner, panoramic observation of the area to be monitored is not possible. This figures in a situation where only one display screen is available. Nevertheless, if more display screens are available, the situation remains unsatisfactory. Thus, a target may for instance disappear from the field of vision of a TV camera and fail to appear in the field of vision of another TV camera.

The system according to the invention does not have this drawback and is characterized in that an array of TV cameras is provided for at least substantially simultaneously visualizing each azimuth direction of the environment .

A compact embodiment of the invention is characterized in that the array of TV cameras is arranged in one plane to cover an at least substantially 360 degree field of view.

A further advantageous embodiment which precludes the occurrence of blind angles is characterized in that each TV camera in the array of TV cameras has an aperture such that each azimuth direction can be monitored with at least one TV camera. In this context it is possible to give each TV camera its own aperture, for instance on the basis of a desired resolution in a certain azimuth direction. In line with an advantageous embodiment of the invention, all TV cameras have similar apertures, which simplifies the merging of sub-images from adjacent TV cameras.

In actual practice it is found to be impossible to align the TV cameras with such accuracy that the images from adjacent TV cameras form a continuous picture. A further advantageous embodiment of the invention is therefore characterized in that the apertures of adjacent TV cameras slightly overlap at least in azimuth direction.

A still further advantageous embodiment of the invention is characterized in that the system includes an image processor for merging the sub-images obtained with the aid of the array of TV cameras into one panoramic image. In effect, this is quite simple, provided that the sub-images are stored in the image processor in the form of matrices of N rows in elevation direction and M columns in azimuth direction. Merging then merely involves a once-only readjustment, in an alignment procedure, of the relative row numbers so that horizontal rows continue and the column numbers run consecutively.

A possible problem during alignment is constituted by the overlap in azimuth direction. A still further advantageous embodiment of the invention is therefore characterized in that the image processor is also designed to define one new pixel from two corresponding pixels in overlapping sub- images. This may be realized through point-wise linear interpolation.

A further advantageous embodiment of the invention is characterized in that the display unit is designed to present at least a part of the surrounding panorama, centred around a selected azimuth direction. Thus, a potential target that is just about to disappear from the display in azimuth direction, can simply be tracked by selecting another azimuth direction. For this purpose, the display unit may incorporate a selector device, for instance a rotary knob.

A highly advantageous embodiment of the invention is characterized in that the display unit includes at least one helmet-mounted display and the selected azimuth direction corresponds with a direction that the user of the helmet-mounted display is facing. Such a display affords a view of the surrounding area from a completely darkened room. This is for instance of importance in a vessel constructed to minimize radar reflection, which rules out the conventional navigating bridge or other superstructure. In accordance with the invention, an array of TV cameras is installed on the ship which can be realized without any appreciable increase of the radar cross-section; the navigation bridge is placed below the water line and the helmet-mounted displays enable crew members to observe the surrounding panorama without hindrance. The helmet-mounted display should be equipped with a sensor for determining the direction of view, however, sensors of this type are well-known in the art.

A further, highly advantageous embodiment of the invention is characterized in that the array of TV cameras comprises an even sub-array and an odd sub-array whereby each sub- array is designed to at least substantially simultaneously visualize each azimuth direction of the environment and always one TV camera in the even sub-array and one TV camera in the odd sub-array are identically aligned in azimuth direction, which produces a stereoscopic effect. A still further advantageous embodiment is characterized in that each of the two sub-arrays comprises an image processor for merging the sub-images obtained with the aid of a sub-array into one panoramic image and that a stereoscopic display screen is provided for stereo- scopically displaying at least a portion of the panoramic images, centred around a selected azimuth direction.

A highly advantageous realization of the invention is characterized in that the display unit comprises a helmet- mounted display and that the selected azimuth direction corresponds with a direction that the user of the helmet- mounted display is facing.

Particularly for ship-mounted applications, the aperture of the TV cameras shall be selected to be so wide that a selected elevation direction, for instance the horizon, remains within view, irrespective of the ship's roll and pitch. A still further embodiment of the invention where this constraint need not be considered, is characterized in that the system is provided with stabilization means for retaining the surface on which the array of TV cameras is disposed in a level position.

A thus stabilized image, which moreover allows a narrow aperture, is most suitable for automatic target detection, because the ship's movements no longer affect the apparent position of a target. A still further advantageous embodiment of the invention is therefore characterized in that the system is provided with a target extractor, connected to the image processor, for initiating and maintaining tracks of potential targets and for supplying azimuth and elevation values pertaining to these potential targets.

A well-known drawback of optronic systems is the lack of range information pertaining to a target. A highly advantageous embodiment which obviates this drawback at least partially, is therefore characterized in that the system is provided with two arrays of TV cameras placed one above the other, each array incorporating an image processor for generating a panoramic image and that furthermore two target extractors, connected to the image processor, are provided for initiating and maintaining tracks of potential targets and for supplying azimuth and elevation values pertaining to these potential targets as well as range information on the basis of parallax shift determined for each potential target. Range information is of particular importance for eliminating bird echoes, in that sense that the possibility of a track being generated is ruled out.

The invention will now be described in greater detail with reference to the following figures, of which: Fig. 1A is a schematic top view of an array of TV cameras according to the invention; Fig. IB is a schematic side view of an array of TV cameras according to the invention; Fig. 2 is a block diagram of an non-stabilized array; Fig. 3 is a block diagram of a stabilized array.

Fig. 1A shows a top view of an array of 18 TV cameras 1, positioned to cover a 360 degree field of view, each TV camera having an aperture of 24 degrees, consequently resulting in 18 images, always with a 4 degree overlap. By means of hinge 2, each TV camera 1 is mounted on a foundation ring 3 and is additionally provided with a lever 4 ending in a second hinge 5 which acts in concert with a disk 6, capable of rise and fall motion, thus allowing the tilt of all TV cameras 1 to be adjusted simultaneously.

Fig. IB shows a side view of the same array incorporating the same component parts. Furthermore, a protective hood 7 is shown to cover the array of TV cameras which, at the position of the TV cameras, is transparent to the wavelength in question. For relevant applications, the protective hood may be covered with a material that absorbs electromagnetic radiation and may be shaped such that it is difficult to be detected by a radar apparatus. The TV cameras 1 used are sensitive to visible light or infrared radiation. In the latter case the cameras are of the type that makes chilling down to liquid nitrogen temperature unnecessary, for instance on the basis of prior-art pyroelectric detectors or bolometer arrays.

Fig. 2 is a block diagram of a non-stabilized array in which 18 cameras 1.1,..., 1.18 are connected to an image processor 8 implemented as a high-speed computer provided with DSPs. Every 20 milliseconds, image processor 8 receives 18 sub-images in the form of image matrices consisting of rows and columns. Each sub-image has a horizontal aperture of 25 degrees; the main function of the image processor 8 is to generate, every 20 milliseconds, one image with a horizontal aperture of 360 degrees from these 18 sub-images. A secondary function of the image processor 8 is to periodically display the relative alignment of the 18 sub-images, for instance after overhaul and by means of a measuring grid placed around the array of TV cameras. Alignment errors comprise a possible translation in horizontal and vertical direction as well as a possible rotational error. Rotational errors can be corrected per sub-image in a manner known in the art by rotating the image matrix corresponding to the sub-image, such that horizontal lines in the measuring grid are also observed to be horizontal. Translations in vertical direction can be simply corrected by slightly adjusting, per sub-image, the row numbers of an image matrix. Even though this method will cause several image lines to be left out at the top or at the bottom of a sub-image, it is still considered acceptable. Translations in horizontal direction, however, upset the continuity of the overall image and shall consequently be corrected. This is relatively easy to realize because in azimuth direction the sub-images afford a certain degree of overlap. In effect, this means that for two adjacent sub-images, a number of columns are identical, which can be easily demonstrated by image processor 8 with the aid of the alignment grid. The actual merging of the sub-images then involves the rotation of each sub-image through a fixed angle, the adjustment of the row numbers for each sub-image and lastly, selection or interpolation for corresponding columns. In this respect, there is a slight preference for interpolation, since it improves the signal-to-noise ratio to some degree. The actual merging of the sub-images in horizontal direction then merely comprises the renumbering of the columns. If a sub-image has for instance 1024 pixels in azimuth, no more than 820 pixels, numbered 1,2..820, may remain after merging, depending on the correction. The first pixel of the contiguous image is then designated number 821. i.e. all column numbers are raised by 820.

After initial alignment it is known how many columns the merged image comprises. In case of for instance a ship- mounted system, the azimuth direction corresponding to each column number will then also be known. If a certain direction is to be scanned, then for instance 1024 columns centred around the desired direction of view can be written to a buffer storage 9 and subsequently be presented on a display 10 in a manner well-known. For selecting the direction of view, a selector device 11 is connected to the image processor 8. An additional possibility is to implement display 10 as a helmet-mounted display and the selector device as a prior-art helmet-mounted angle sensor. Such a system creates the illusion of looking straight through a ship's hull or a tank's armour-plating, which makes it possible to for instance eliminate the navigating bridge with its considerable radar cross-section. Besides, other data, e.g. radar data, can simply be added to the image thus obtained, which may considerably enhance the weapon deployment decision process.

It is also possible to replace the 18 TV cameras presented in Fig. 1 by 18 pairs of TV cameras, each pair generating a stereophonic image. By means of an image processor, the images from the left TV cameras of all pairs can be merged into one panoramic image; this likewise obtains for the images from the right TV cameras of all pairs. This method yields two panoramic images from which a direction may be selected; the image in that direction can be displayed panoramically in a manner well-known, for instance by means of a helmet-mounted display. If the direction is determined by the direction that the user of the helmet-mounted display is facing, this results in a most spectacular and useful system for operating a vehicle or vessel without the possibility of physically observing the surrounding area.

Fig. 3 shows a block diagram of a stabilized array with the 18 cameras 1.1,...1.18 mounted on a prior-art stabilized platform, so that ship movements, with the exception of the ship's heading, no longer affect the system. The 18 cameras 1.1,...1.18 are again connected to an image processor 8 which for instance generates, every 20 milliseconds, one image with a horizontal aperture of 360 degrees from the 18 sub-images, this in line with the array described with reference to Fig. 2. In a stabilized array, a target will every 20 milliseconds appear on practically the same spot on the display screen. Particular importance attaches to a point target looming up on the horizon, such as a missile or patrol vessel. For the detection of point targets, image processor 8 is connected to a point target extractor 12 which compares the intensity of each pixel with the intensity of adjacent pixels and decides on the basis of thereof if a point target is actually observed. Point target extractor 12 assigns the detected point targets to a tracker 13 only after the presence of a point target has been ascertained in a (string of images) number of sequentially generated images. Additionally, point target extractor 12 may comprise other prior art components to reduce the false alarm probability, for instance edge detectors which detect the edges of for instance man-made structures or clouds and preclude the detection of point targets in those areas. For each detection, tracker 13 initiates a tentative track which, after a certain number of detections, changes into a confirmed track to be subsequently presented on a display unit 10.

Besides detections, point target extractor 12 is also provided with data on the ship's heading 14 with which, on the basis of modified column numbers, an alteration of the ship's course can be simply corrected such that it seems as if the target does not move as a result of the alteration of course.

Birds flying in close proximity to the array show all the features of a point target and will inevitably cause the generation of tracks. To verify the presence of birds, a second array may advantageously be added to the existing array; the second array may be positioned above or underneath the existing array. The tracks supplied by both arrays are passed to one display unit 10 for multi-colour presentation. A bird flying in the proximity of the arrays will show a significant parallax difference, causing the tracks to run parallel without any overlap. There is no visible parallax difference for a point target appearing on the horizon; such a target is presented as a single track in a different shade of colour. The parallax difference may of course also be ascertained with a suitable computer program. This precludes tracks caused by birds from appearing on the display screen.

Claims

1. System for visualizing an environment with the aid of TV cameras and at least one display unit, characterized in that the system comprises an array of TV cameras for at least substantially simultaneously visualizing each azimuth direction of the environment.

2. System as claimed in claim 1, characterized in that the array of TV cameras is arranged in one plane to cover an at least substantially 360-degree field of view.

3. System as claimed in claim 2, characterized in that each TV camera in the array of TV cameras has an aperture such that each azimuth direction can be monitored with at least one TV camera.

4. System as claimed in claim 3, characterized in that each TV camera has an at least substantially identical aperture.

5. System as claimed in claim 3 or 4 , characterized in that the apertures of adjacent TV cameras slightly overlap at least in azimuth direction.

6. System as claimed in claim 5, characterized in that the system includes an image processor for merging the sub- images obtained with the aid of the array of TV cameras into one panoramic image.

7. System as claimed in claim 6, characterized in that the image processor is also designed to define one new pixel from two corresponding pixels in overlapping sub- images. _╬¢ΓÇ₧_{> o}ΓÇ₧ 9/00982

12

8. System as claimed in claim 6 or 7 , characterized in that the display unit is designed to present at least a portion of the panoramic image, centred around a selected azimuth direction.

9. System as claimed in claim 8, characterized in that the display unit incorporates a selector device for selecting an azimuth direction.

10. System as claimed in claim 8, characterized in that the display unit includes at least one helmet-mounted display and that the selected azimuth direction corresponds with a direction that a user of the helmet-mounted display is facing.

11. System as claimed in claim 4, characterized in that the array of TV cameras comprises an even sub-array and an odd sub-array, whereby each sub-array is designed to at least substantially simultaneously visualize each azimuth direction of the environment and always one TV camera in the even sub-array and one TV camera in the odd sub-array are identically aligned in azimuthal direction.

12. System as claimed in claim 11, characterized in that each of the two sub-arrays comprises an image processor for merging sub-images obtained with the aid of a sub-array of TV cameras into one panoramic image.

13. System as claimed in claim 12, characterized in that a stereoscopic display screen is provided for stereo- scopically displaying at least a portion of the panoramic images, centred around a selected azimuth direction.

14. System as claimed in claim 13, characterized in that the display unit comprises a helmet-mounted display and that the selected azimuth direction corresponds with a direction that the user of the helmet-mounted display is facing.

15. System as claimed in claim 6 or 7, characterized in that the system is provided with stabilization means for retaining the surface on which the array of TV cameras is disposed in a level position.

16. System as claimed in claim 15, characterized in that the system is provided with a target extractor, connected to the image processor, for initiating and maintaining tracks of potential targets and for supplying azimuth and elevation values pertaining to these potential targets.

17. System as claimed in claim 15, characterized in that the system is provided with two arrays of TV cameras placed one above the other, each array incorporating an image processor for generating a panoramic image and that furthermore two target extractors, connected to the image processor, are provided for initiating and maintaining tracks of potential targets and for supplying azimuth and elevation values pertaining to these potential targets as well as range information on the basis of parallax shift determined for each potential target.