WO2008015545A2

WO2008015545A2 - Imaging telesurveillance system and method for monitoring an area to be supervised, in particular an airport area

Info

Publication number: WO2008015545A2
Application number: PCT/IB2007/002210
Authority: WO
Inventors: Giorgio Pelosio
Original assignee: So.G.Aer S.P.A.
Priority date: 2006-08-02
Filing date: 2007-08-01
Publication date: 2008-02-07
Also published as: ITTO20060576A1; WO2008015545A3; EP2055103A2

Abstract

The invention relates to an imaging telesurveillance system and method (1) for monitoring an area to be supervised, in particular an airport area, comprising means (10, 20, 30, 40, 50) for obtaining a plurality of panoramic images (LVi, IRi, MMi, W1i, W2i) of said area and a display device (300). The system according to the invention is characterized by comprising means (100) for merging together at least two images of said plurality of panoramic images (LVi, IRi, MMi, W1i, W2i) into a global image (101) which provides a larger view of said area, wherein said global image (101) can be represented visually on said display device (300).

Description

IMAGING TELESURVEILLANCE SYSTEM AND METHOD FOR MONITORING AN AREA TO BE SUPERVISED, IN PARTICULAR AN AIRPORT AREA

DESCRIPTION

The present invention relates in general to an imaging telesurveillance system and method for monitoring an area to be supervised.

More in particular, the present invention relates to an imaging telesurveillance system and method for monitoring a delimited and restricted area wherein there is movement of aircraft, surface vehicles, assistance vehicles and staff, such as an airport area.

Nowadays, safety is an essential aspect for everyday life. This is particularly true when it comes to constantly coordinating the activities of a plurality of aircraft, surface vehicles, assistance vehicles, maintenance staff, etc., operating within the airport area on a daily basis. To this end, surveillance systems have been conceived which use a plurality of sensors collecting information about the positions of transports and people operating within the airport area, said information being then sent to an operator.

However, accidents still occur rather often in airport areas, and may cause serious damage to things and people. Several systems are known in the art which allow to supervise an entire airport area, in particular the movements of vehicles and people operating therein.

For example, patent US 6,246,320 describes a security and surveillance system for airports which comprises a plurality of video cameras and illumination means, such as an infrared illuminating, for supplying visual information, audio sensors for detecting noise, thermal sensors for detecting smoke or fire in the airport area, and a GPS system for recognizing the different transports and watching the movements thereof. The images from the video cameras are shown to an operator on a screen subdivided into several windows, or else on as many screens as the number of video signals to be displayed.

For example, patent application EP 0 785 536 describes the use of a plurality of millimeter- band radars for an airport surface traffic monitoring system.

For instance, patent application No. US 2003/0169335 describes a surveillance system based on the use of a plurality of sensors for monitoring transports parked or moving in an airport area, such as aircraft and commercial vehicles. However, the images sent to the operator and displayed on a screen relate to single transports, thus not providing a global view of the airport area.

For example, patent application EP 0 714 082 describes the use of a plurality of radars, among which a ground radar and another radar for locating flying aircraft, for the purpose of monitoring traffic within an airport area.

For example, patent No. EP 0 883 873 describes a system for guiding traffic on an airport using a radar, which transmits a synthesized image of the situation of the traffic in the airport to a screen located in a control tower. The synthesized image must however be interpreted by the operator, since the screen displays data in the form of abbreviations and numbers. Therefore, the data displayed on the operator's screen cannot be read easily by anybody, in that the presence of a skilled operator is required for its interpretation. Known systems for monitoring a delimited and restricted area therefore generally require that the data analysis be performed by a skilled operator, both because the data is displayed in a disaggregated manner on one or several screens and because the data is displayed as alphanumerical symbols to be interpreted and understood.

The present invention therefore aims at overcoming the above-mentioned drawbacks by providing an imaging telesurveillance system and method for monitoring an area to be supervised, in particular an airport area, which shows the information received from a plurality of subsystems interacting within a single representation context.

It is a further object of the present invention to provide an imaging telesurveillance system and method for monitoring an area to be supervised which allows an operator to select the most adequate type of visual representation for the environmental conditions of the supervised area. It is a further object of the present invention to provide an imaging telesurveillance system and method for monitoring an area to be supervised which can provide a larger view of the supervised area.

It is a further object of the present invention to provide an imaging telesurveillance system and method for monitoring an area to be supervised wherein the operator is not required to attend special training courses in order to be able to operate the system.

It is a further object of the present invention to provide an imaging telesurveillance system and method for monitoring an area to be supervised which can provide a real-time view of the supervised area.

These and other objects of the invention are achieved by the method and the system as claimed in the appended claims, which are intended as an integral part of the present description.

According to the present invention, the following describes an imaging telesurveillance system and method which allow to find and identify objects and/or people in space through a combination of sensors even in adverse visibility conditions, such as ground fog, darkness, mist and air turbulence. The system according to the invention has been conceived for the purpose of optimizing the performance of an airport monitoring system by including a data merging system capable of displaying images coming from different types of sensors (visible-light video camera, infrared video camera, radar image) as well as further visual information shown in association with said images on a common video output device . The main object of such an integration algorithm is to provide the operator with a single RGB video channel which can highlight useful information dynamically depending on a particular operating condition. In practice, the system according to the invention defines an interactive output image control system capable of highlighting the information pertaining to specific monitoring needs from time to time. The merging system has five different input video channels:

- one visible-light RGB channel, equivalent to three monochromatic channels;

- one monochromatic channel from an infrared video camera;

- one monochromatic channel from a radar subsystem;

- two possible monochromatic channels generated synthetically and containing information, mostly text information, to be overlaid on video, e.g. the flight number of an aircraft taking off and/or landing.

In short, said merging consists in mapping the bands of the infrared, radar and synthetic channels in the visible spectrum by using an adaptive criterion for the perception of the operator of each of them. The behaviour of such a system is defined by a set of primary control parameters consisting of three real variables, which have values in the range [-1, 1], and five binary variables, the value of which can only be either 1 or 0. Through these flags, it is possible to enable or disable each of the visible-light, infrared, radar and synthetic channels.

The primary control parameters allow to define dynamically the behaviour of the merging procedure. The primary control parameters are calculated by a second subsystem implemented at hardware or firmware level based on specific design choices. This subsystem receives a set of secondary control parameters consisting, in the simplest case, of the status readings relating to an operator's pointing system, e.g. a joystick. This allows to simulate the behaviour of the operator who, in real time, generates the levels of merging in the image as needed by controlling the joystick.

Also, the invention includes a real-time control system which, based on appropriate information obtainable from the input channels, provides additional secondary parameters (besides those manually entered by the operator) which can be exploited by the primary parameter calculation system in order to implement some kind of automatism. For example, environmental visibility conditions can be estimated from the visible-light channels, thus optimizing the brightness level of the infrared and radar channels. The system according to the invention includes a merging subsystem which, starting from primary control parameters, generates the merged image to be displayed. It is a local (pixel- by-pixel) algorithm which can be easily implemented in vectorial form, thus obtaining a computational advantage from a direct utilization of SIMD structures being present in the calculation units on board of DSP cards used for implementing the large view system. The computational scheme for the implementation of the merging algorithm has been created in the Matlab® environment. The above objects will become apparent from the detailed description of the method and system according to the invention, with particular reference to the annexed figures, wherein:

- Fig. 1 illustrates a basic diagram of a system for monitoring an area to be supervised, in particular an outdoor airport area, according to the invention;

- Fig. 2 shows a structure of a phase of formation of a visible-light panoramic image; - Fig. 2a shows a possible arrangement of three video cameras in a portion of the supervised area; - Fig. 2b shows a structure of a phase of formation of an infrared panoramic image; Figs. 3a, 3b and 3c respectively show a test image and two images in which there is a

"trapezoidal" vertical perspective distortion;

Figs 4a, 4b and 4c respectively show a test image, an example of "pincushion" distortion (center image), and an example of "barrel" distortion (right image) on the test image of Fig. 4a;

Figs. 5a e 5b respectively show a detail of a target and an example of a target image distorted by the superimposition of a perspective distortion (tilt) and a "barrel" type geometric distortion; - Figs. 6a and 6b respectively show the distorted target and the detail of Figs. 5a and 5b filtered by means of a Gauss-Laguerre function;

Fig. 7 shows a square search window for finding the local maximum points of a transformed image;

Figs. 8a, 8b and 8c respectively show an image representing a non-distorted target, an image representing the target of Fig. 8a with tilt and "barrel" distortion, and a correction of the distorted target of Fig. 8a;

Figs. 9, 10 and 11 show a structure of a phase of formation of a radar panoramic image in visible light;

- Fig. 12 shows a reference scene generated by a phase of formation of a reference scene image;

Fig. 13 shows an image containing operational symbols generated by a phase of formation of an image containing operational symbols;

Fig. 14 shows an image obtained by overlaying the image of Fig. 12 on the image of Fig. 13; - Fig. 15 shows an exemplifying diagram of the image merging method according to the invention;

- Fig. 16 shows a control device used by an operator for controlling the image merging method;

- Fig. 17 shows a block diagram of an image merging method according to the invention; - Fig. 18 shows a graphic visualization of the commands entered by the operator through the control device of Fig. 16. Referring to Fig. 1, it shows a basic diagram according to the invention of an imaging telesurveillance system 1 for monitoring an area to be supervised, in particular an airport area.

System 1 comprises a phase 10 of formation of a visible-light panoramic image LV₁, a phase 20 of formation of an infrared panoramic image IR₁, a phase 30 of formation of a radar panoramic image MM₁, a phase 40 of formation of an image Wl₁ of a reference scene of the supervised area, and a phase 50 of formation of an image W2, containing operational symbols of fixed or moving elements of the supervised area.

The images LV₁₃IR₁₃MM₁₅Wl i,W2, output from the respective formation phases 10,20,30,40,50 are sent to a merging module 100 for merging the images

LV₁₅IR₁₅MM₁₅Wl _l5W2_l5 which merges together one or more of said images

LV₁₅IR₁₅MM₁₅W I₁, W2_l5 in order to supply a global image 101 to the operator.

The integration of images LV₁₃IR₁₅MM₁₃Wl _l5W2, is controlled by an operator through a suitable control device 20O₃ so that one or more of said images LV₁₃IRi₅MM₁₃W I₁, W2_l3 are selected and displayed on a display device 300.

Referring now to Fig. 2, it illustrates in detail the structure of phase 10 of formation of a visible-light panoramic image LV₁.

Phase 10 of formation of a visible-light panoramic image LV₁ comprises three visible-light video cameras 12, which shoot a portion of the supervised area, in particular an airport area.

The adoption of three visible-light video cameras 12 allows to cover an azimuth angle of

90°, as shown in Fig. 2a₃ wherein three visible-light video cameras 12 cover an azimuth angle of 30° each.

For the entire supervised area to be under the control of visible-light video cameras 12, it is preferable to use at least twenty-four visible-light video cameras 12 in order to cover an angle of 360° and to provide a stereoscopic panoramic image. However, the number of visible-light video cameras 12 may be smaller or greater than twenty-four: the important thing is that the entire airport area to be monitored is adequately covered by the range of visible-light video cameras 12. Images 12a shot by three video cameras 12 are treated in succession by three geometric correction modules 14, which provide geometrically corrected images 14a, and by two radiometric correction modules 16, the functions of which will be described in detail later on.

A radiometrically corrected image 16a is then sent to a visible-light panoramic image composition module 18, which cuts and pastes images 16a and outputs a visible-light panoramic image LV; in RGB format.

Referring now to Fig. 2b, it illustrates in detail the structure of phase 20 of formation of an infrared panoramic image IR.

Formation phase 20 comprises three infrared video cameras 22, which shoot a portion of the delimited and restricted area to be monitored. As in the case of the visible-light video cameras 12, it is preferable to use at least twenty- four infrared video cameras 22; however, a smaller or greater number of infrared video cameras 22 may be used, as long as the entire area to be supervised is covered by infrared video cameras 22. Images 22a shot by three video cameras 22 are treated in succession by three geometric correction modules 24, identical to the geometric correction modules 14, which output geometrically corrected images 24a, and by two radiometric correction modules 26, identical to radiometric correction modules 16, the functions of which will be described in detail below. Image 26a, radiometrically corrected, is then sent to an infrared panoramic image composition module 28, which cuts and pastes the images and outputs a monochromatic infrared panoramic image.

Still with reference to Fig. 2, the following will now describe in detail the function carried out by geometric correction module 14, this description being also applicable to geometric correction module 24 of Fig. 2b by simply taking into account that the geometric correction of module 24 is applied to infrared video cameras 22.

As previously stated, in a preferred embodiment of the invention system 1 according to the invention comprises a stereoscopic panoramic set of 24 (2 x 12) RGB type visible-light video cameras 12 capable of returning a video stream with a resolution of 720 x 576 pixels. Images 12a from visible-light video cameras 12 are generally distorted by two kinds of geometric distortion: a vertical perspective distortion due to the orientation of visible-light video camera 12 ("tilt distortion"), and a geometric distortion due to the groups of lenses making up the optics of visible-light video camera 12 ("optical distortion"). As far as vertical perspective distortion is concerned, visible-light video cameras 12 are obviously installed in system 1 at a certain height from the ground, which implies a downward vertical orientation (positive tilt) that produces a "trapezoidal" geometric distortion of image 12a.

Figs. 3b and 3c show an example of "trapezoidal" distortion relating to a non-distorted test image, shown in Fig. 3 a.

Figs. 4b and 4c respectively show the effect of two of the most common types of geometric distortion caused by optics, i.e. "pincushion" distortion and "barrel" distortion, while Fig. 4a shows a test image having no distortion.

"Pincushion" distortion, shown in Fig. 4b, is typically produced by teleobjectives operating with high zoom values. It consists of a deformation of horizontal and vertical lines toward the center of the image. "Barrel" distortion, shown in Fig. 4c, may be considered to be the dual of "pincushion" distortion; it is typical of wide-angle lenses and consists of a distortion of horizontal and vertical lines toward to edges of the image.

Typically, "pincushion" distortion is less visible and disturbing than "barrel" distortion. Most commercial photo retouching programs allow both to apply such deformations to a digital image and to correct an existing one through the application of appropriate inverse distortion models ("spherize").

Optical distortion is a regular function of image coordinates, with only a few low-order spatial derivatives being non-negligible. Therefore, the distortion function can be well approximated by means of polynomials or of low-order rational functions . Distortion correction procedures based on bicubic interpolation algorithms can reduce the low-pass effect which is typical of linear interpolation procedures, such effect limiting the actual attainable resolution.

In system 1 according to the invention, optical and perspective distortion correction is effected not only by the necessity of improving as much as possible the verisimilitude of the images perceived by an operator, but it is also a fundamental step required for aligning with the utmost precision the images of video cameras belonging to adjacent sectors before putting them together in order to form a panoramic image ("mosaicing"). In general, given a distorted image and with no information about the type of acquisition perspective, the distortion compensation procedure consists of a series of attempts made by the operator by changing the characteristic parameters of inverse distortion models. Said attempts are guided by the semantic contents of the image and by the operator's experience. Since in most cases total distortion is the result of the superimposition of several types of deformation, the compensation procedure is very subjective and generates different outcomes depending on the operator's skill and on the particular image taken into consideration.

However, according to one aspect of the invention, it is possible to make the compensation procedure become objective by estimating a distortion model on the basis of appropriate test images, called "targets". In substance, it is necessary to define:

- a distortion model;

- a target for the optical tests, according to which the model can be estimated;

- an algorithm for estimating the distortion model; - a compensation algorithm capable of returning a non-distorted image by applying the inverse model to the acquired image as described below.

As far as the distortion model is concerned, it is possible to adopt two distinct types of models: parametric models and non-parametric models.

Since parametric models involve the unquestionable drawback that a specific distortion function must be selected in advance (i.e. dangerous hypotheses must be made initially about the distortion nature), system 1 according to the invention adopts a non-parametric model consisting in the definition of the shift caused by distortion for each characteristic point, or vertex, of the target.

A non-parametric model also allows for the simultaneous treatment of distortions derived from the superimposition of distinct phenomena, such as the superimposition of a perspective distortion on a geometric distortion, e.g. "pincushion" type.

The position of the vertexes on the non-distorted target can be determined from the intersections of lines suitably drawn on the target itself. In this regard, the target image is built by means of an equally spaced grid of (vertical and horizontal) black segments on a white background; the intersections of these lines are the target vertexes. The grid is centered on the target. In practice, the target is printed on a screen which is located at a distance from the video camera to be characterized which is at least greater than 25 times the focal length of the objective lens, in order to make focusing mechanically possible and to avoid introducing into the image any marked field curvatures or other optical aberrations being present in the nearby field. The dimensions of the screen and its distance from the objective are such that each line on the acquired digital image is approximately one pixel thick. In the case of system 1, it is preferable to work at the greatest feasible distance, due to the non-negligible size of the multiple support of visible-light video cameras 12. The image of the non-distorted target has a resolution of 720 x 576 pixels. The target has been so conceived that between two adjacent rows (and columns) there are exactly 15 white pixels (row and column period of 16 pixels).

It follows that the target determines 36 x 45 = 1620 vertexes, the positions of which on the non-distorted image are known a priori. Based on these positions, the original vertexes can be ordered univocally according to a lexicographic order (the vertexes of the upper first row from left to right first, then the vertexes of the second row, and so on).

In general, the position of the i^th vertex on the distorted image will be altered according to a shift vector di=(dxi, dyi). A distortion model consists of a matrix Δ having dimensions (1620, 2) and containing all shift vectors arranged according to the same lexicographic order as described above. This order implicitly specifies which vertex refers to which shift vector included in the matrix Δ.

If the extra-axial chromatic aberration of the optics is negligible, the distortion model can be calculated starting from the image luminance channel (following a transformation from RGB representation to LUV or HSB representation) and can then be applied simultaneously to the three colour planes R, G and B through the correction algorithm described below.

An algorithm for calculating the matrix Δ starting from a distorted image of the target will now be described in detail. Of course, since the correction procedure is carried out once for all in off-line mode, the algorithm will have to be defined by privileging precision and robustness over other aspects relating to computational complexity. Figs. 5a and 5b show a distorted target image, the vertex of which must first of all be determined. The difficulty of such a problem of "template matching" is highlighted in the detail, shown in Fig. 5b, of the distorted target shown in Fig. 5a. It can be seen that the vertexes are identified by lines which are not necessarily parallel to the edges of the image. Please note also the effect of the antialiasing filter typically included in any optics supplying a CCD ("Charge-coupled device") digitalization system. De facto, a filter of this kind is always present along the optical path upstream of a dichroic prism, i.e. an optical system which divides light into its three RGB colour components, performing the function of limiting the energy of high spatial frequencies. Spatial frequencies higher than those that can be detected entering the prism-CCD system would produce visible aliasing (and artifacts). The light entering the detection system must therefore be limited through said anti-aliasing optical filter, so that it cannot produce higher detail than that that can be detected. This filter is low-pass type, and its construction depends on the performance of the CCD detector. Low-contrast optical targets can be created as an alternative, the lines of which are suitably smoothed in the perpendicular direction for the purpose of reducing aliasing. The above-mentioned "template matching" problem can be solved by using functions of the Gauss-Laguerre Transform (GLT) (see for example the article written by Alessandro Neri and Giovanni Jacovitti, "Maximum Likelihood Localization of 2-D Patterns in the Gauss-Laguerre Transform Domain: Theoretic Framework and Preliminary Results", IEEE Transactions on Image Processing, vol. 13, no. 1, January 2004). The GLT transform is based on a family of orthogonal circular harmonic functions characterized by specific radial profiles (i.e. Gauss-Laguerre functions) through which it is possible to define transformation filters having interesting "template matching" properties (invariance with respect to scale factors or rotations). The generation of an appropriate FIR type numerical filter essentially depends on four parameters: a) m, n: dimensions of the Gauss-Laguerre filter (both numbers must be even); b) k: radial order of the filter; c) q: angular order of the filter; d) sigma: scale factor in pixels of the Gaussian function generating the Gauss-Laguerre base.

Several simulations have shown that satisfactory filtering results can be obtained with m=n=32, k=0, q=4, sigma=13, by up-sampling the distorted image by a variable factor between 4 and 6.

The up-sampling process allows to estimate the vertexes on the distorted image with better precision. The filtering result can be seen in Fig. 6a. The local maximum points of the transformed image identify the target vertexes. One procedure for finding said maxima provides for analyzing the whole image through a

13x13 square search window (this dimension is compatible with the minimum expected distance of a vertex from the edges of the image).

The difficulty in searching said maxima is given by the presence of maximum luminance values shared among contiguous groups of 2 or 4 pixels arranged as a horizontal or vertical rectangle or as a square, as shown in Fig. 7. Another problem is the possibility of false positives due to edge effects visible in the detail of Fig. 6b.

At the end of the search procedure, a suitable procedure for matching the found vertexes to those being present on the non-distorted target (based on a clustering criterion of the "nearest neighbour" type) allows to order the matrix Δ as described above.

Finally, each element of the matrix Δ must be scaled by taking into account the up- sampling factor.

Thus, when the estimation process is completed, the elements of the matrix Δ will contain the coordinates of the vertexes in the form of fractional numbers, the adopted unit of measure being one pixel in the native resolution.

The distortion correction algorithm will now be described, together with an improved formulation for representing the distortion model which employs two scattered matrixes.

The distortion correction algorithm provides a representation of the matrix Δ which is less compact but more effective in terms of computational complexity. Once the distortions due to the optics of the video camera have been characterized, it is possible to define correction algorithms. As previously described, said characterization is made by using an optical target which allows to identify the shift of some characteristic points, or vertexes, in the acquired image.

As a basis for any correction algorithm, an effective representation of the data used must be provided for both computational and performance reasons.

Given a grayscale image having dimensions (W, H), the observed luminance values of the single pixels can obviously be represented through a matrix (W, H) of suitably normalized values. It is however appropriate, for subsequent processing, that also the abscissa and ordinate values of the single pixels of the image are allocated in matrixes having dimensions (W, H). In this particular instance, the element X(ij) of the matrix X will contain the abscissa of the pixel lying in the i'¹ row and j⁴ column, while the element Y(i,j) of the matrix Y will contain the ordinate thereof. Assuming that such values are normalized between 0 and 1, the following will be obtained:

X(ij) = (i-1) / (N-I) Y(Ij) = G-I) / (N-I) where N = max(W,H) and i=l ...W, j=l ...W

The vertexes of the image can also be represented in a similar manner. It is therefore assumed that they are spaced evenly, i.e. positioned on a subgrid within the original target (non-distorted) with Wc elements per row and Hc elements per column. In this situation, the normalized abscissas and ordinates of the vertexes can be represented through matrixes Xc' and Yc', each having dimensions (Wc, Hc).

Referring now to Fig. 8a, it shows a grid having dimensions 720x576, wherein the vertexes corresponding to the intersections between horizontal and vertical lines are spaced evenly by 16 pixels in both the horizontal and vertical directions. Thus, the vertexes belong to a subgrid having dimensions Wc=720/16=45 and Hc=576/16=36.

The optical distortion characterization algorithm is therefore based on the detection, in the acquired image, of the shifts of the vertexes along the horizontal and vertical axes. A matrix Xc is thus obtained, the generic element Xc(m,n) of which contains the position actually occupied on the abscissa (in the acquired image) by the element in the m^th row and in the n¹¹¹ column of the vertex subgrid. Likewise, a matrix Yc will contain the positions actually occupied by the vertexes on the ordinates.

Referring now to Fig. 8b, it shows a plausible distortion (with "tilting" and "barrel" type distortion) of the target of Fig. 8a, which clearly highlights how the vertexes tend to shift on both axes. It should be remembered that said values may also be fractions, in that the estimate of the shifts of the vertexes is made on the up-sampled image. For the image to be corrected, it is necessary to characterize the shift of all of its pixels, not only of its vertexes. In this regard, it is possible to use interpolation algorithms by working on both axes in a decoupled manner. The problem is therefore to estimate the function f_x:5R²→ % the output value f_x(ij) of which is the shift of the generic pixel in the i^th row and

f³ column along the X axis. Likewise, it will be necessary to estimate the function f_y:9ϊ — >

SR for the shift of each pixel along the Y axis. Each of these two functions can be estimated by solving an interpolation problem, since the values of said functions are known in certain points, i.e. those previously estimated for the vertexes. For example, for the vertex in the position (m,n), the value of the shift on the abscissa will be Xc(m,n) - Xc'(m,n). It is therefore clear that the accuracy of the interpolation result increases with the number of vertexes. There is however a limit due to the optical resolution beyond which the vertex can no longer be discriminated with sufficient accuracy.

By using an even grid for the vertexes, since such is also the output grid (i.e. the one of the image pixels), it is possible to use a simple and fast bilinear or spline interpolation algorithm. Simulations have shown that, for this type of images, an interpolation through a bicubic spline provides very accurate results even in high distortion situations. The knowledge of the shifts of each pixels along both axes characterizes the optics distortion completely. It is then possible to proceed to the correction algorithm by making the following considerations. For each pixel in the i^th row and j^th column of the actual image (non-distorted), the luminance value Z(i,j) is observed; this value is observed in said image on the abscissa X(i,j) and on the ordinate Y(ij); the actual position of said pixel in the original image (non-distorted) can be estimated as X'(i,j)=X(ij)-f_x(i,j) and Y'(ij)=Y(i,j)-f_y(ij). At this point, since shifts can be fractional, it is necessary to re-sample the image. This means that an interpolation must be carried out in order to know the luminance values in the actual positions (indicated by matrixes X and Y) starting from the knowledge of the luminance values in the points indicated by matrixes X' and Y'. The unevenness of these latter matrixes makes the interpolation algorithm more complex than the previous one. For example, it is possible to use QhulPs cubic interpolation method based on Delaunay triangulations (see for example the web page http://www.qhull.org). Fig. 8c shows the correction made on the target of Fig. 8a distorted as in Fig. 8b. As shown, a satisfactory distortion correction is obtained on the whole image. Any inaccuracies on the edges of the image are not very significant, in that the correction step is followed by the image cropping step, which precedes the panoramic pasting step. A subdivided interpolation may even be performed, i.e. applied to sub-blocks of the image, aiming at compensating for any sharp distortion specifically due to "tilting" phenomena. Referring back to Fig. 2 and Fig. 2b, the function of radiometric correction module 16 will now be described in detail, this description being also applicable to the radiometric correction module 26 of Fig. 2b simply by taking into account that the radiometric correction of module 26 is applied to infrared video cameras 22.

In system 1 according to the invention, the preliminary distortion correction, obtained through geometric correction modules 14,24, is assumed to be accurate enough to define the pasting boundary with pixel precision and to minimize coma effects. The images obtained from video cameras 12,22 must in fact be joined together in order to form a panoramic image; this step is carried out separately for the visible-light image, for the infrared image, and for any left/right stereoscopic planes.

Moreover, if video cameras 12,22 work properly, the horizontal gradients of the images on the left and on the right of the pasting line will be very similar in direction, their width being proportional to the average brightness of the two partial images (if the value does not reach the upper saturation "shoulder" in the sensitivity characteristic of the CCD).

In this case, it is possible to define a "piecewise" function by which the images on the left and on the right can be multiplied, such that: a) the function value is 1 on the central vertical line of each image, which therefore is not altered; b) on each horizontal line, the function must level out the values of those pixels of the two joined images which are located on the right and on the left of the boundary, thereby bringing them to the mean value of the same pixels on the original images; c) the function is linear in the horizontal direction; d) the function must be calculated and applied separately for each RGB plane in visible and infrared light; e) since the shooting conditions are relatively stationary, the function is preferably updated every 4-10 frames in order to reduce the computational load; f) the construction of the proposed function requires the exchange of a few columns of pixels among the processors which process adjacent angular sectors. In formulas, assuming that M and N are even integers, and that: • Z₁ (x, y); 0 < x < M ; 0 < y < N are the values of the pixels of the image on the left;

• Z₂ (x ', y '); 0 < x ' < M; 0 < y ' < N are the values of the pixels of the image on the right;

• f_ (x, y); M /2 < x ≤ M - 1; 0 < y < N is the function for amplifying the image on the left of the boundary; • f₊ (x ', y '); 0 < x ' < M /2 - 1; 0 < y ' < N is the function for amplifying the image on the right of the boundary;

• m(y) = — — — ² — ; 0 < y < N is the horizontal average of the pixels on

the boundary, by definition the following will be obtained:

. f₊(M/2 -l,y') = l; 0 ≤ y' < N;

The interpolating functions for each ordinate will therefore be given by the following straight line equations:

The same interpolating functions will then have to be multiplied by the original images for the purpose of performing a colorimetric compensation, according to the formulas: Z₁(X, y) = f_(x,y)I_λ(x,y); M/2 ≤ x < M; 0 ≤ y < N /₂(^/) == /₊(χ',/)/₂(^χ',/); O ≤ X ' < M/2_; o ≤ y < N .

Reference will now be made to Figs. 9 and 10 in order to illustrate in detail the structure and operation of phase 30 of formation of a visible-light radar panoramic image MM.

Fig. 9 shows a millimeter-band radar 32 shooting an image 32a of the supervised area, which is then transmitted to a module 34 which carries out a "ray tracing" procedure, i.e. a procedure used for finding the incidence of the luminous rays hitting objects being present in a tridimensional scene, so as to generate a radar panoramic image MM which will then be integrated with the images from visible-light video cameras 12 and from infrared video cameras 22. In a preferred embodiment, every second radar 32 generates a matrix of echoes, encoded as

8-bit integers and having actual dimensions of 1019 (range) x 4096 (azimuth), after having removed some meaningless data.

The method adopted for projecting the polar coordinates which are typical of a 2-D radar

(range-azimuth) on an image is referred to as "cosecant transformation" and is schematically recalled and integrated in Fig. 10.

The processing carried out by said "cosecant transformation" method is subdivided into two steps: an off-line step and a real-time step.

The first step, which is carried out off-line during the calibration of system 1, creates a matching map between each range cell of radar 32 and one pixel of the panoramic image through the definition of ground points X(θ, R) and by executing the "ray tracing" procedure on a focal plane PF. These algorithms can be run by generic-use computing machines, such as electronic computers.

On initialization, the matching map is then encoded and loaded in video processors, i.e. a specific server provided for that purpose. In particular, the map is physically constructed as an array of structures ("array of records" in PASCAL and FORTRAN 90/VAX).

Due to requirements relating to memory occupation and to real-time processing parallelism, each sector of a panoramic video camera 35 has its own separate map.

The size of each sector array is equal to the number of range cells having ground points

X(Θ,R) within the sight field of video camera 35. Each element of the array includes one record containing the addresses (or displacements) of the range cell to be projected, the addresses of the corresponding pixels of the projected image plane, and one or more auxiliary fields for selecting the visualization of the range cells or specifying advanced properties thereof (e.g. colour, size of the spot on the video plane, etc.).

The second step, carried out in real time, consists in the creation, with the help of the address map, of the sector radar image plane MM to be subjected to the integration procedure. The most exacting task performed during this step is the calculation of memory read/write addresses.

The off-line calibration step is carried out when the system 1 is installed and during periodic re-calibrations. The fundamental operations of the calibration step are as follows: georeferencing system 1, finding the ground points X(Θ,R) to be reproduced on the image, projecting the ground point X(Θ,R) on focal plane PF.

As to georeferencing system 1, the relative positions of radar 32 and of each sector video cameras 35 are measured and referred to a fixed Cartesian reference system. The position reference points are the nodal points of objectives and antennas. As to measuring the sweep start azimuth of radar 32, as well as the actual focal length and the field angle of the optics, this data can be partially obtained during the optical distortion correction procedure. With reference to Fig. 10, the step of finding the ground points X(Θ,R) to be reproduced on the image calculates for each range cell of radar 32 the Cartesian coordinates in the chosen fixed reference system of the corresponding ground point X(Θ,R) located at the indicated azimuth θ and distance R. If radar 32 is located at a height h_r above ground, the distance of X(Θ,R) from the base of radar 32 will be:

R_eff = ^R² -h_r ² for R > h_r , whereas the azimuth θ is unchanged. For R < h_r, the cell will not be re- projected. Therefore, all objects S(Θ,R) located at the same distance and azimuth from radar 32 will be projected on the same point X(Θ,R). Due to the bidimensionality of radar 32, this ambiguity cannot be eliminated.

In a preferred embodiment, the final result of this step is a map (array) containing the coordinates (x, y, z) of 1019 x 4096 ground points X(Θ,R). Each point X(Θ,R) is re-projected through the "ray tracing" procedure on the focal plane PF of each sector video camera 35 comprising it within its field angle.

The equation of the straight line passing through X(Θ,R) and through nodal point F of the objective is then calculated. The intersection between this straight line and focal plane PF provides the desired point P(Θ,R) .

It is important to keep the position of the centroids and the orientation of the focal planes consistent among all shooting sectors during the optical and perspective distortion correction step, in order to obtain a good overlay of radar and video planes and to make it easier to paste final panoramic image MM.

The 3-D absolute coordinates of P(Θ,R) must finally be converted into whole 2-D pixel coordinates on the focal plane P(u,v) .

The set of Q points P(Θ,R) -> P(u,v) provides the map necessary for the transformation. With reference to Fig. 11, concise vectorial designations are used for defining: n : unitary length vector (versor) indicating the direction perpendicular to focal plane PF, with output sense toward the sensor side; u : versor aligned with the local abscissa axis of focal plane PF, assumed to be parallel to the ground plane; v : versor aligned with the local ordinate axis of focal plane PF, so that u,-v,n define a dextrorotatory tern of local Cartesian coordinates on focal plane PF;

P₀ : geometrical center of focal plane PF;

F : focal center of the objective of video camera 35;

/: focal length of the objective; z : unitary versor of optical axis 37 of the objective, passing through focal center F and geometrical center P₀ of the focal plane and directed out of the camera:

P : generic projection P(Θ,R) on the focal plane of X(Θ,R) , The following analytical relationships are established: - coplanarity relationship between P₀ and P, so that n»P=n»P₀ , where • designates the 3-D scalar product;

- position of the focal point: F = P₀ -fa for an objective focused to infinite distance.

- projection P on the focal plane of the corresponding point X(Θ,R) through the radius passing through F :

JT t

- local coordinate system, with origin at P₀:

- local coordinates of the projection P: u (P- P₀) . u

P= (P-P₀)^v

The local coordinates have to be further scaled and quantized with respect to the dimensions of one pixel (by taking into account the "aspect ratio" of the pixels, i.e. the ratio between base and height of the pixel) in order to obtain the pixel index \p_u p_v] to be associated with the generic point X(Θ,R) on the focal plane.

Referring now to the second step of the "cosecant transformation" method, the map (or list) of addresses for each sector (or video camera), hereafter referred to as A, contains Q records, each containing at least the following fields:

• azimuth address kg on the radar image, which in a preferred embodiment is within the range (0,4095).

• range address k_r on the radar image, which in a preferred embodiment is within the range (0,1019). • address p_x of the abscissa of the pixel re-projected on the image plane.

• address p_y of the ordinate of the pixel re-projected on the image plane.

The length Q of the address list depends on the angular sector taken into account by the selection of the range cells. It must be considered that several range cells may be projected on the same pixel by the "ray tracing" procedure, especially for far-away objects. It is therefore necessary to choose a strategy. It will be supposed below that the pixel of the radar image plane is attributed the greatest value of all radar echoes projected thereon. being I the matrix containing the re-projected radar image and being V the matrix of the raw radar data, the creation of the radar image plane can be summarized by the following algorithm, expressed in pseudo-code:

I=O; for q=l to Q,

I(A(q).px,A(q).py) = max(I(A(q).px,A(q).py),V(A(q).kr,A(q).kθ)); end Tests carried out have shown that the radar sweep start angle ( θ=0 ) must be calibrated accurately. However, the angles indicated by the radar are very stable among successive sweeps. Therefore, the re-projection maps need not be updated in real time during the operation of system 1. Also, the exact scale factors of the radar must be verified by introducing a sufficient number of scatterers in known positions (at least three per angular sector of the camera). The same scatterers are also very useful for the fine alignment of the image planes of panoramic video camera 35.

Referring to Fig. 12, the following will illustrate a reference scene Wl generated by a phase 40 of formation of a reference scene image.

Since the position of the sensors, consisting of video cameras 12,22,35 and radar 32, is fixed to the ground, system 1 stores a reference image providing the contours of the objects forming reference scene Wl, as well as additional reference points such as building profiles

41, road edges 42 and fences. The dimensions of the areas and buildings being present in the scene are known; all important points are georeferenced. Reference image Wl is built on the basis of digitalized drawings and/or high-resolution photos. Referring to Fig. 13, an image W2 containing operational symbols is illustrated which is generated by phase 50 of formation of an image containing operational symbols.

In order to assist the operator, the system provides symbols overlaid on the telesurveillance image, which are acquired by receiving position/identification signals, e.g. from GPS

(Global Positioning System), multilateration or IFF (Identification Friend or Foe) type systems. Such symbols may for example represent commercial vehicles, aircraft, fixed points, etc. Fig. 14 shows an image 61 obtained by overlaying images Wl and W2.

In summary, system 1 provides, for each panoramic image generated (two in the stereoscopic vision case, one in the monocular vision case): three visible-light RGB colour planes, one IR monochromatic plane, one monochromatic plane generated by cosecant transformation of the radar image, one monochromatic plane of the reference image, and one monochromatic plane W2 of operational symbols.

All of the above-listed monochromatic channels have a standard representation with 8 -bit integers and dynamics between 0 and 255. Below, for mere mathematical synthesis reasons, said representation will be normalized between 0 and 1, thus coinciding with a discrete representation of 256 possible real values evenly distributed between 0 and 1.

Therefore, for each pixel there are seven variables inputted to the image merging module

100.

The final visualization uses three RGB or equivalently encoded image planes. Only point functions are taken into consideration, which act by combining only homologous pixels of the images.

The merging problem can therefore be formally solved by creating a map associating 7 input variables, all subjected to 8-bit quantization (256 levels) with 3 output variables quantized within the same interval: y_R(m,n) x_R (m, ή)

0 ≤ m < M y_G(^m>ⁿ) — J x_o (m, ή) , x_m (TO, 77), x_MM (m, n), x_Wχ (m, n), x_Wi (m, n) 0 ≤ n < N x_B(m,n) where:

[y_R{m,^ή) y_G(m,n) y_B(m,n)] ^ ^ ^ _{RGβ planes tπmsmitted tø display} device 300,

\ Lx ^ΛJ^\m,^>ri) ^/ x_r ^Cz(^Vm,'n)^/ x ^BJ^\m,'ή) ^/j\ are t .,he t .,hree co ,lour p ,lanes f _rom vi .si ._tb ,le- _Λli.g _Λh_Mt vi .d,eo camera 12, xm vⁿ> ⁿ⁾ is the plane generated by infrared video camera IR, x^{MM Vn> n)} represents the image plane generated synthetically through cosecant transformation of the polar map transmitted by the millimeter radar (MM), ^w^ ' ' is the plane generated by reference scene Wl ; x w₂ (^\in ' yι^j) j_s ^₆ plane generated by operational symbols W2.

The merging must highlight the details detected by each sensor, while not overloading the human observer. All seven image planes input to image merging module 100 are represented by one byte and are considered to be normalized to one.

Fig. 15 shows a block diagram 90 comprising an adaptive control module 98, a module 99 for calculating primary control parameters, and the merging module 100, already shown in Fig. 1. Panoramic images LVj₅IRj₅MMi₅WIi and W2;, obtained from formation phases 10,20,30,40,50, respectively, are input to modules 98 and 100. The input of module 99 consists of the inputs generated by the operator through control element 200 and by secondary control parameters output by module 98. The output of module 99, consisting of the primary control parameters X₃Y₅Z and of enable/disable flags α(.), is input to merging module 100, which provides global image 101.

With reference to Fig. 16, the adopted control element may be a joystick 200 provided with a shaft 210 and a key 211 which allow the joystick 200 to move along four analog axes 201,202,203,204 (respectively corresponding to the axes x,y,z,w)₅ a plurality of programmable keys 205, and a "hat switch" 206. A joystick of this type is, for example, the "Microsoft Sidewinder®".

In practice, the secondary control parameters consists of the status readings relating to axes 201,202,203,204 and to programmable keys 205, which readings can be easily carried out by using (with Microsoft operating systems) the primitives included in the Direct Input library. It is therefore possible to associate each programmable key 205 with a binary flag enabling the corresponding channel, whereas the readings of the joystick position along the axes 201, 202, 203 are the secondary control parameters from which the primary parameters X, Y and Z are obtained. It is important to underline that the distinction between primary and secondary control parameters allows to design a sufficiently flexible system which permits to implement useful adaptive control procedures easily. Furthermore, it may be useful to calculate the primary parameters from the readings on the joystick axes in a non-linear manner, e.g. by implementing a "logarithmic slider" system; to this end, when studying "human factors" it will be necessary to assess the ergonomics of such a system based on psychovisual tests carried out on different operators. According to a possible definition, "human factors" are a set of physiological and psychological features of the human being intended as a being capable of processing information. Fig. 17 shows the merging method implemented in image merging module 100. Since the method only concerns one pixel at a time, the diagram includes the following inputs:

(Zv) Q

~nat representing one RGB pixel of the geometrically and radiometrically compensated visible-light image;

IR representing one monochromatic pixel of the geometrically and radiometrically compensated infrared image;

MM representing one monochromatic pixel of the 2D radar video projected on the reference image plane; Wl representing one pixel of the reference image;

W2 representing one pixel of the plane of operational symbols.

(out)

The output is a pixel in RGB format designated -³ .

The diagram also shows the product images Wl -W2 and IR-MM, also having dynamics between zero and one. In the following diagram, the luminance image and the chrominance image are calculated for each image.

Each of images LV, IR₅ MM, Wl and W2 to be merged together may be either enabled or disabled for the next merging process. This is done through appropriate flags, which either set the respective image to zero (i.e. black) or leave it unchanged. Therefore, each image should hereafter be considered to be the outcome of the following updates:

• RGB theme of image LV

luminance of image IR IR «- α^(ir) • IR;

• luminance of image MM MM <- α^(mm) • MM;

• luminance of image Wl Wl «- α^(wI) « Wl;

• luminance of image W2 W2 ÷- α^(w2) • W2.

The values of the flags, represented by the various coefficients α^{( )}, may only be either 0, for a disabled image, or 1, for an unchanged image.

The following will describe the process for mapping the images LVi, IRi, MM;, Wl; and

W2j in order to obtain a global image 101 on the RGB theme.

The image LV may be mapped to the RGB theme in two different manners: by natural image or by grayscale.

When the image LV is mapped by natural image, the output RGB tern coincides with the tern of the pixel q to be mapped, represented by the column vector q^ O) .

— nat

—nat G^{h)

When the image LVj is mapped by grayscale, a matrix A> (Iv) ^; (block 110) is used so as to obtain, in the column vector q^{h) , the mapping of the images LV; on a gray value related to natural luminance L (Iv) .

Remembering that Y is a primary control parameter with values between -1 and 1, the combination of the previous functions gives the following overall mapping of the images LVi to the RGB theme: for 0 <Y < 1, the natural pixel LV is obtained; for -1 < Y < 0, the natural pixel becomes shaded toward grayscale as Y approaches the value -1. This can be represented by the RGB tern of the column vector q :

i.e. (blocks 112, 114 and sum node 116 of Fig. 18), minφ,Y)A (Iv) _G(/v)

From the above expression it can be easily verified that, if the matrix A (⁽Iv) ' is such as to ensure output RGB values always comprised between 0 and 1 , then also the elements of q^{!v) will be always comprised between 0 and 1. In conclusion, the overall mapping can be re- written as follows:

where I is the 3x3 identity matrix.

, (lv) A possible computational acceleration can be obtained when the matrix — maps exactly

. (Iv) the luminance L^{( v} in -^&&y , thus obtaining:

3 l?1 ^v) ₊ [min(0,Y)(q (Iv) (Iv)

so that the calculation of formula 1.a can be accelerated as follows:

T (Iv) where — is a column vector wherein all components contain the previously calculated value L^> = rR^ -^H gG^ ₊ bB^_{i τhe ∞emχάeπts r} « ₀,30, g « 0,59, b ~ 0,11 correspond to the luminances of the (pure) red, green and blue colours, respectively, for r + g + b = 1. Another form of computational acceleration, which produces one more multiplication but two less additions, and which is more useful in view of the final optimization of the merging formula, is the following: qf^v) = (l + min(0,Y))q⁽ _n ⁱ;_t> - min(0,Y) L^ _ ^

The matrix A is used for mapping the images LV; on a gray value related to luminance ϊβ^ . Since all coefficients of q must be equal, a property must hold according to

which all rows of A are equal.

It is also necessary that the mapping of A ^■* generates a vector q , the elements of

which are all comprised between 0 and 1. For this to be true, since each element of — q⁽g'r^va⁾y is

the linear combination of the values R^(lv) G^(lv) B^(lv) , obviously comprised between 0 and 1, through the row coefficients of A^(Iv) , it is apparent that a property must hold according to which the coefficients of each row of A are comprised between 0 and 1 and the sum thereof is smaller than or equal to 1.

In particular, if natural luminance L^(lv) were to be mapped exactly in q⁽'^v) , independently

of the optimization conceived in formulas Lb or l.c, the matrix A5^IV) should be the following:

In this latter case, it is also interesting to note that, as a result of an exact mapping in luminance L^ with the algorithm in question, the luminance L⁽/^v) of the output q^(Iv) remains constant as the control parameter Y changes. In fact, the following is obtained:

min(0,Y) [r g b] b)} = L^(lv)

(Iv)

Note that the possible matrix — is static, i.e. it does not change during the execution of the merging operations; therefore, the respective coefficients must be calculated during the

compilation phase. The matrix — A^(lv) is actually determined when studying "human factors".

Still with reference to Fig. 17, the process of mapping the images IRj and MMi to the RGB theme will now be described.

The images IR; and MMj can be mapped to the RGB theme in two different manners: by chromatic separation or by grayscale.

(irm) Chromatic-separation mapping uses a matrix A (block 118) in order to obtain, in the firm) column vector -^cro , the mapping of the image IR; to a first colour, the mapping of the image MMj to a second colour, and the mapping of possible overlay cases to a third colour:

R (irm) cro IR

(irm) _ _c (irm)

—cro ^ cro MM R (irm) IR - MM

A formula of this kind has been suitably chosen for preventing problems of image spatial superimposition and to compress the dynamics of the output image within specified limits without having to resort to expensive and visually unpleasant algebraic saturation operations.

^(^irm) Grayscale image mapping uses a matrix — ² (block 120) in order to obtain, in the q(irm) column vector ^~gray , the mapping of the images IR, and MMj to a gray value related to the intensity of the respective images: τ> (inn) l^vgray IR

(irm) _ r^. (inn) _ A (^irm)

-gray ^ gray MM ■_□ (inn) ■"gray IR - MM

Remembering that X is a primary control parameter having values between -1 and 1, the combination of the previous functions gives the following overall mapping of IR and MM to the RGB theme: for O≤X≤l, the mapping of chromatic separation only is obtained; for - l≤X≤O, chromatic separation becomes shaded toward grayscale as X approaches the value

(inn)

-1. The above is represented by the RGB tern of the column vector -¹ :

(l + min(0,X)) —q< c™ro > ~ min(0,X)

i.e. (blocks 122, 124 and sum node 126 of Fig. 18),

IR IR

;^irm) (irm)

5i = (l + min(0, X))Aj MM - Im_n(O₅X)Af¹⁰ MM

IR - MM IR - MM

Λ <-^irm) A (^iπΗ)

From the above expression it can be easily verified that, if the matrixes — ^l and — ² are such as to ensure output RGB values always comprised between 0 and 1 , then also the

(inn) elements of -¹ will be always comprised between 0 and 1. In conclusion, the overall mapping can be re- written as follows:

IR q^(irm) ₌ J₄ ⁽^⁾ ₊ [_min(05X₎(AS^irm) - A₂ ^im) MM IR - MM

(2)

(irm)

As previously described, the matrix —^l is used for mapping the image IR; to a first colour, the image MM; to a second colour and possible overlay cases to a third colour. In a preferred embodiment, a selection has been made to fulfill the following particular criteria: • a pure pixel IR (IR=I, MM=O) is preferably mapped to red;

• a pure pixel MM (IR=O, MM=I) is preferably mapped to blue;

• the total overlay of two pixels IR and MM (IR=I, MM=I) is preferably mapped to green;

• intermediate cases are treated with edge smoothing functions in order to fulfill the preceding cases.

By mapping a pure pixel IR to pure red, a pure pixel MM to pure blue and total overlay to pure green, the respective luminances (and thus the respective perceptions) would be different. In fact, the luminances of said primary colours would be r = 0.30 _^ b ≤ O.l l _^ g = ⁱoy _^ respectively. In order to prevent this problem from occurring and to generalize the mapping, it has been chosen to define the reference luminance in advance for each single case. Below, therefore, L_r, L_b and L_g will designate the luminances obtained in the cases of pure pixel IR, pure pixel MM and total overlay, respectively. For this generalization to be verified, it will thus be necessary to fulfil additional criteria: in the case of the pure pixel IR, any luminance portion not covered by red ( ,L. r -r ) is obtained by mapping a portion of the pixel IR to green; in the case of the pure pixel MM, any luminance portion not covered by blue

U -b

( ) is obtained by mapping a portion of the pixel MM to green;

• in the case of total overlay, any luminance portion not covered by green ( ^s ) is obtained by mapping the pixel IR and the pixel MM to red and blue proportionally. A possible method for fulfilling all of the above criteria is obtained through the following equations:

R«^m) = [(l-MM)- IR]+ -(L. -^β) • MM - IR r + b

B^ = I(I-IR)-MM]₊ (L. -g) - MM - IR r + b thereby obtaining the matrix

1 0 r + b

A (^irm) _ L_r-r L 1-L_r-L_b £≥l - g g

^CfQ

Therefore, the luminance obtained for chromatic mapping is always given by:

-IR)+(L_b-MM)+[(L_g-L_r-L_b)-MM-IRJ

thus obtaining luminance L_r in the case of the pure pixel IR, luminance L_b in the case of the pure pixel MM, and luminance L_g in the case of total overlay.

A (^irm)

Another important condition to be fulfilled by the coefficients of the matrix —^l for the purpose of avoiding expensive saturation operations to return within the dynamics limits, is

that the resulting coefficients

must always be comprised between 0 and 1. This sets some constraints as to the allowable values of L_r, L_g and L_b.

Sufficient conditions for said constraints to be fulfilled are: ^{r ~ r} ; ^{s ~} ; b≤L_b≤l. (L_r+L_b)≤l

Below the simplest choice will be referred to, according to which ^{r ~r}, ^g , ^{b ~} • By doing so, the possibility of mapping residual components of the pixel IR and of the pixel MM to green is excluded and the following reference matrix is obtained: 1 0 -1

A (^irm) — 0 0 1

0 1 -1 arding matrix — A (inn) Reg ² , this is used for mapping the pixel IR and the pixel MM in the q(iπn) vector ^~gray to a gray value related to the intensity of the respective images. Since all q(iπn) coefficients of ^~gray must be equal, a property must hold according to which all rows of

— ² are equal.

The simplest way to determine this relationship would be to add together the intensities of the images IRj and MMj. The following matrix would then be obtained:

1 1 0

(inn)

A₂ 1 1 0

1 1 0

A matrix of this kind would however require a saturation operation, in that the resulting

coefficients

, ^gray would not always be comprised between 0 and 1. An alternative solution is:

1 1 -1

£±2 ^~ 1 1 -1

1 1 -1

(irm) (inn)

Note that the matrixes — A^{U l} and — Λ ² are static, i.e. they do not change during the execution of the merging operations; therefore, the respective coefficients must be

calculated during the compilation phase. The actual determination of the matrixes — ^l

and — A^{( 2irm)} takes place when studying human factors.

The following will now describe the process of mapping the images WIj and W2i on the global image 101 with RGB theme.

The channels WIj and W2; can be mapped to the RGB theme in two different manners: by chromatic separation or by grayscale. J. (WW)

Chromatic-separation mapping uses a matrix -¹ (block 128) to obtain, in the column

vector -^cro , the mapping of Wl to a first colour, the mapping of W2 to a second colour, and the mapping of possible overlay cases to a third colour:

(ww) _

Also in this case, the above formula facilitates overlay situations and improves behaviours at the dynamics limits.

(ww) . (ww)

Grayscale mapping uses a matrix — ² (block 130) to obtain, in the column vector ^~gray , the mapping of the images WIj and W2j to a gray value related to the intensity of the respective images Wl;,W2i:

"

The combination of the above functions gives the following overall mapping of the images WIi and W2; to the RGB theme.

Remembering that Z is a primary control parameter having values between -1 and 1, for O≤Z≤l, the mapping of chromatic separation only is obtained; for -l≤Z≤O, chromatic separation becomes shaded toward grayscale as Z approaches the value -1. All this is

(ww) represented by the RGB tern of the column vector ^₁

(ww) _ (ww)

%

i.e. (blocks 132, 134 and sum node 136 of Fig. 18)

(ww) (ww)

From the above expression it can be easily verified that, if the matrixes — ^l and — ² are such as to ensure output RGB values always comprised between 0 and 1, then also the

(ww) elements of — ^l will be always comprised between 0 and 1. In conclusion, the overall mapping can be re-written as follows:

q(ww)

2

(3. a)

Since the pixels Wl and W2 have been conceived as binary ones (e.g. text pixels with values 0 or 1), a possible computational acceleration of formula 3. a can be obtained by simplifying the grayscale mapping calculation:

Wl ww) ) q;^TO) = (l + πώi(0, Z))Af W2 -min(0,Z)δ^ (ww Wl - W2

(3.b)

. (ww) where -° is a column vector in which all components contain the previously calculated

value ^δ»™' = ™*(^W1' ^W2) , so that ^δ°™' " k '!

Furthermore, should the images Wl and W2 be mapped to the same colour, the overall mapping could be reduced to:

i.e. q(w) _{= δ}(ww) ^(ww) _{+ min(}o_{5 Z)}[a(ww) _i] }

(3.C) a (ww) where ~³ is the column vector the components of which define the colours for chromatic mapping, and the symbol - designates a column vector containing only Is.

A (^irm) A (inn)

In principle, the very same considerations made for the matrixes -¹ and — ² also

( (.wwww); L , ( (.ww) apply to the matrixes — ^l and — ² , possibly only requiring some changes in the characteristics of the colours whereto a pure pixel Wl, a pure pixel W2 or a total overlay thereof should be mapped. However, as mentioned, the default condition taken into account herein is that of binary channels Wl and W2, the computational optimization of which leads to avoiding using the

Λ (™) matrix — ²

(ww)

This makes defining the matr Πi ^•Yx —— A 1 ^l easier as well. Assuming that:

• the pure pixel W (Wl=I, W2=0) is mapped to a colour represented by an RGB tern in

(W) T-P (WW) _r (ww) r>(ww) ]^r . the column vector -wi ^~ L^κwi "wi ^βwi J '

• the pure pixel W2 (Wl=O, W2=l) is mapped to a colour represented by an RGB tern

(ww) _ [τj (ww) n(vw) D{WW) F . in the column vector -W2 - L^KW2 ^W2 %2 J '

• total overlay (Wl=I , W2=l) is mapped to a colour represented by an RGB tern in the

,(ww) __ L (WW) p (ww) D(WW) ]¹ . column vector -^W12 L^W12 ^UW12 ^βW12 J '

It should be noted that there are no intermediate conditions, since the pixels Wl and W2 are binary. At this point, it is apparent that a matrix fulfilling the above requirements while

also ensuring that the coefficients

are always comprised between 0 and 1 is the following:

^(ww)

Below, a matrix — ^l having the following features will be referred to:

• the pure pixel Wl is mapped to orange;

• the pure pixel W2 is mapped to cyan;

• total overlay is mapped to white. As a result, the matrix employed will be: 1 0 0

A (WW ) 0 .392 1 - 0.392

0 1 0

A (W) _Λ (WW)

Note that the matrixes — ^! , — ² are static, i.e. they do not change during the execution of the merging operations; therefore, the respective coefficients must be

(ww) . (WW) calculated during the compilation phase. The matrixes Ai — ² are actually determined when studying human factors.

Still with reference to Fig. 17, the following will now describe the algorithm used for merging the various images, all of which have already been mapped to the RGB theme.

This merging algorithm includes the steps of: fading image LVj, fading images MMj and

IRi, chromatic deviation of image LVj, merging images LV;, IR; and MMj, overlaying images Wl; and W2;, final optimization.

As to the step of fading image LV;, the RGB signal corresponding to the mapping of image

LV, which forms the main background of the merged pixel, is faded increasingly as the Y axis (Y=O) and the edges (|X|=1) are approached. This is obtained in the RGB tern of the

(Iv) (Iv) vector -² by starting from the vector -¹ . which in turn is calculated in block 138 with one of formulas La, Lb or Lc:

R₂ ^1V)

= max(l -|X|,|Y|)q .₁ ⁽(Iv) q₂ ^lv) =

B (Iv)

(4)

, (lv)

Having previously verified that the elements of -¹ are always comprised between 0 and

(Iv)

1 , it is clear that this will always be true also for the elements of ~² . As to the step of fading images MM; and IRj, the RGB signal corresponding to the mapping of images MM; and IRj is faded with isolevel curves corresponding to squares having sides parallel to the adjustment area (XY plane) and the fading increases as the intersection of

(irm) the X and Y axes approaches. This is obtained in the RGB tern of the vector -² by (irm) starting from the vector -¹ , which in turn is calculated in the block 140 with formula 2:

(irm)

Having previously verified that the elements of -i are always comprised between 0 and

(inn)

1, it is clear that this will always be true also for the elements of -²

(Iv)

As to the step of chromatic deviation of signal LV, signal LV, faded in -² through block 138, is "chromatically deviated" as a function of signal IR and MM. As described above,

(irm) the latter is associated with -² through formula 5. The deviation takes place through a

O (Iv) matrix — (block 142) under the constraint that, in the output RGB tern represented by

(Iv) (Iv) the vector -³ , the signal shall keep the same luminance as that of -² but shall change

(irm) colour according to the characteristics of -² (sum node 144): for example, in the case of an IR-only signal mapped to pure red, the signal LV will become "redder" while still maintaining the same luminance.

It is important to underline that the unchanged luminance constraint does not ensure that

(Iv) the elements of -³ will be always comprised between 0 and 1. Therefore, overflow or underflow phenomena may occur which will be described when referring to the step of merging images LV;, IRi and MM;. o(iv) q . (-Iv) q .O- v)

The matrix — is used in (6) to obtain in -³ a chromatically deviated signal -² as a

(inn) (Iv) function of -² , under the constraint that the same luminance as that of -² shall be

kept. The structure of a matrix — Ω^(lv) fulfilling such a constraint may be the following:

It is therefore necessary to determine the various coefficients of this matrix. To do so, the

_L(lv) _Q (lv) luminance ³ of the vector -³ must be determined and then set to be equal to the

luminance

:

which is:

Lr = [(r-gp-bp)R₂^]₊[(g-rγ-b_γ)G₂^]₊[(b-rβ-gβ)B^]₊L?

T (Iv) _ T (Iv)

By setting to zero the quantities between round brackets, the constraint ^{^3} IS fulfilled, thus obtaining the values of the coefficients P , ^, P as a function of the reference luminances r, g, b. The following default matrix can then be used:

g b r + b r + g b 1 - 1.422 - 0.129

Ω^(lv) = 1 b + g r + g - 0.426 1 - 0.129 r g 1 b + g r + b - 0.426 - 1.422 1 _I (Iv)

Note that the matrix — is static, i.e. it does not change during the merging operations; therefore, the respective coefficients must be calculated during the compilation phase. The

matrix — Ω^(lv) is actually determined when studying human factors.

As to the step of merging images LV;, IR; and MMi, the merging of signals LV, IR and MM processed so far is obtained first by reducing the chromatic deviation of LV increasingly as, when moving parallel to the X axis, the edges are approached (|X|=1). With reference to blocks 146 and 148 and to sum node 150, in the intermediate RGB tern

(W) (Iv) represented by the vector -^aux this fades the chromatically deviated signal (-³ ) toward

(Iv) the signal LV (-² ), which has only been faded, not deviated:

— q a^(lu^vx⁾ = aux X qf + (HxDa (Iv)

(inn)

The faded signal - q² of IR and MM is always added to the signal q through sum node

152, for the purpose of increasing the luminance, and thus the perception, when there is a signal IR and/or MM. Therefore, a first version of the signal derived from the merging of

. (o«t) the images LV;, IRi e MMi is obtained in the RGB tern of the vector -¹

i.e. using formula (6), q(out) +,Jv)]

and then, applying further compaction, q(out) ₌ q(lv) ₊ |_{j +} JJ₁ _|_χ|)^(lv) ]}q(irm) (out)

As mentioned previously, the elements of the signal -¹ are not necessarily comprised

(Iv) between 0 and 1, because this is not true for -³ and also because during these latter steps

(inn) the signal -² was added.

It is therefore necessary to carry out a saturation process (block 154) in order to obtain the

(out) definitive merging of the images LV;, IR; and MM; into -²

(out)

2

(out) _ (out)

% = max(θ,minfe^out),l))

B (out)

(7) where the bold format of - and - indicates a column vector comprising all Os or Is, respectively.

As to the step of overlaying the images Wl; and W2j, the images WIj and W2j are essentially conceived for overlaying additional information. Rather than merged with the q^(out) preceding signal -² , it is more correct to say that images Wl; e W2; are overlaid on it

(ww) after having been mapped in the vector -¹ through one of formulas 3.a, 3 ,b or 3.c.

(ww) _α(°^ut)

This overlay is done by totally replacing "l w wiitthh - "² for Z<0, whereas for Z>0 there

(ww) (vuϊ) is a crossed fading leading to fading -¹ completely and using only -² as Z=I is approached. This algorithm must however include a "chroma-keying" function in order to prevent the overlay when channels Wl and W2 contain no information. This means

(ww)

"holing" the black colour in the signal -¹ , which in such a case will not be overlaid. One of the various possible ways to obtain this kind of overlay, leading to the final merging

(out) signal -³ , is the following (blocks 156 and 158):

q(out) _ {[l -max(0,Z)]q;~⁾ +max(0,Z) qf" , _ifrmx (Wl,W2) > 0 isT otherwise

(8.a) (ww ) n^(out) Having previously verified, in particular through (7), that the elements of -¹ and -² are always comprised between 0 and 1, it is clear that this will always be true also for the

(out) elements of -³

Since images WIi and W2j have been conceived as binary ones (e.g. text pixels with values 0 or 1), a computational acceleration of the calculation of formula (8. a) is possible, mainly in order to avoid the conditional test: τ> (out) ^3

(out) _ G (out)

% = [1 - max(0, Z)] q;^w) + [l - δ<^w> (l - max(0, Z))]qf^ut)

R (out) °3 (8.b)

using the previously calculated value ⁵^ ^{~ max}(^{W1> W2}) , _So that ^δ°^{W ~} ^⁰' ¹^ .

. (out)

As to the final optimization of the merging algorithm, the signal -³ is the final pixel of the merging processing of images LV, IR, MM, Wl and W2.

(out)

The signal -³ can be calculated by starting from formula 8 and substituting all the other previously determined formulas in cascade, from formula 1 to formula 7. The following final merging formula has been optimized as a direct function of the input channels. With reference to the options available for formula I₅ formula 3 and formula 8, hereafter we will use the luminance L ^v for the grayscale mapping of the images LV; (formula l.c) in combination with the binary images WIj and W2; mapped to different colours (formulas 3.b and 8.b):

Wl qfMHzN (ww) W2 - min(0, Z)δ<^w) + [l - δ_o ^w) (l - max(0, Z))]max(θ, min(c_o , l)) Wl - W2

, (9) where

Ci = [l + min(0, Y)] max(l - |x|, | Y|)

C₂ = min(0, Y) max(l - |x|, |Y|)

C₃ =max(|x|,|Y|){l + [(l-|x|)Q^(lϊ)]}{A_bS^tan) + [min(0, X)(A?^m) -A₂ ^irm))]}

using the same previously calculated values of L and ° The merging calculation algorithm therefore includes the following steps:

1) calculation of the luminance L ^v as a function of the current value of the RGB tern of LV;

2) calculation of the coefficient ° as a function of the current value of channels Wl, W2; 3) calculation of the coefficients ^ci and °² as a function of control parameters X, Y only;

4) calculation of the matrix — C ³ as a function of control parameters X, Y and of the

_Ω(lv) ^ (inn) A (inn) constant (and predefined) matrixes — _? — ⁱ _} —^{2 •}

5) calculation of the vector -° as a function of the coefficients calculated during the preceding steps and of the current value of channels LV, IR, MM;

(out)

6) final calculation of -³ as a function of the coefficients calculated during the

A (irm) preceding steps, of control parameters Z, of the constant (and predefined) matrix — ¹ , and of the current value of the pixels Wl, W2.

The calculation schemes developed within the context of image integration have a dual purpose: to carry out a qualitative evaluation of the performance of the integration algorithms as the primary control parameters change; and to verify the correctness and efficiency of the vectorial functions, ready to be subsequently implemented in the vectorial hardware.

An important feature of the computational scheme is pixel-by-pixel processing, which does not require any processing around each pixel and thus facilitates the vectorial implementation in graphic cards, e.g. Matrox Odyssey^® type cards. The main characteristics of the specific integration procedure employed relate to the mapping of bands IR, MM, Wl and W2 in the visible spectrum by using an adaptive criterion for the operator's perception of each of them. The algorithm maps said bands according to what is specified in the main relationship. Fig. 18 shows a control panel 250, e.g. obtainable in the Matlab environment, through which it is possible to control the levels of integration of the images. Control panel 250 comprises an upper square panel 251, which shows the X and Y axes of control element 200 used by the operator, and a lower horizontal slider 252, which represents the Z axis of control element 200. Both of these controls include a circle 253 representing the current coordinates X, Y, Z. Each click will change the coordinates, resulting in an update of the merged image. By clicking on said figure outside said controls, the coordinates will be reset to the central position, thereby simulating a return spring possibly included in control element 200. Note that joystick 200 can be calibrated to have a default value of X, Y, Z other than zero in its central position, as well as to have different sensitivity to the movement of joystick 200 along the different axes at full scale. Under the two main controls, there are also boxes 254 relating to the single video channel enable flags. A flag is set to ^{α =} 1 when the respective box is checked, or to ^{α =} 0 when the respective box is unchecked.

The controls are changed gradually; below are some types of integration obtained at the geometrical extremes of the various controls, indicated by the respective cardinal points: • X / Y axes (upper square panel 251)

- Center (X=O, Y=O): natural image LV₁;

- N (X=O, Y=I): overlay of the images IRj and MMj, mapped to the respective colours, on the chromatically deviated image LVi; - NE (X=I, Y=I): overlay of the images IRj and MMi, mapped to the respective colours, on the natural image LV;;

- E (X=I, Y=O): overlay of the images IR; e MM;, mapped to the respective colours, on black background;

- SE (X=I, Y=- 1): overlay of the images IRj and MMj, mapped to the respective colours, on the grayscale image relating to the luminance of the channels LV;;

- S (X=O, Y=- 1): overlay of the images IRi and MMj, mapped to the respective colours, on the grayscale image relating to the luminance of the channels LV,, subsequently subjected to chromatic deviation;

- SW (X=- 1, Y=-l): overlay of the original images IR, and MM₁ (i.e. having their acquired luminance) on the grayscale image relating to the luminance of the channels LV₁; - W (X=- 1, Y=O): overlay of the original images IR₁ and MM₁ (i.e. having their acquired luminance) on black background;

- NW (X=- 1, Y=I): overlay of the original images IR, and MM, (i.e. having their acquired luminance) on the natural image LV₁.

• Z axis (lower horizontal slider) - Center (Z=O): total overlay of the planes Wl₁ and W2,, mapped to the respective colours, on the image previously obtained for a given value of X and Y;

- E (Z=I): fading, until complete disappearance, of the planes Wl₁ and W2,, mapped to the respective colours;

- W (Z=- 1): total overlay of the original colour planes Wl₁ and W2, (i.e. black and white) on the image previously obtained for a given value of X and Y.

All of the above-described calculation methods can be executed on at least one electronic computer.

The present invention therefore also relates to an information technology product which can be loaded in the memory of at least one computer and which comprises software code portions for implementing said methods. In this frame, any reference to such an information technology product is intended as a reference to a medium which can be read by a computer and which contains instructions for controlling a computer system for the purpose of coordinating the execution of the method according to the invention. The reference to "at least one computer" is meant to highlight the possibility of implementing the present invention in a distributed and/or modular manner.

It is clear that the above description is provided by way of non-limiting example, and that variations and changes are possible without departing from the protection scope of the invention.

For example, although the invention has been described herein with particular reference to an airport area, it may also of course be applied to any other delimited area to be supervised, such as a harbour, a car park, and so on.

Claims

1. Imaging telesurveillance system (1) for monitoring an area, in particular an airport area, comprising means (10, 20, 30, 40, 50) for obtaining a plurality of panoramic images (LV,, IR₁, MM₁, Wl,, W2[) of said area and a display device (300), said system being characterized by comprising means (100) for merging together at least two images of said plurality of panoramic images (LV₁, IR₁, MM₁, Wl₁, W2^ into a global image (101) providing a larger view of said area, wherein said global image (101) can be represented visually on said display device (300).

2. Imaging telesurveillance system (1) according to claim 1, characterized in that said panoramic images (LV₁, IR₁, MM₁, Wl₁, W2[) are obtained in different frequency bands.

3. Imaging telesurveillance system (1) according to claim 2, characterized in that said frequency bands comprise the frequency band of visible light, the frequency band of infrared light, the frequency band of a radar, and the frequency band of the representation of GPS systems, multilateration systems or IFF systems.

4. Imaging telesurveillance system (1) according to claim 3, characterized in that said frequency band of a radar is the frequency band of a millimeter radar.

5. Imaging telesurveillance system (1) according to claim 1, characterized in that said at least two images of said plurality of panoramic images (LV₁, IR₁, MM₁, Wl₁, W2[) can be selected through a control element (200).

6. Imaging telesurveillance system (1) according to claim 5, characterized in that said control element (200) comprises means (205,206,210,211) for obtaining a linear merging of at least two images of said plurality of panoramic images (LV₁, IR₁, MM₁, Wl₁, Wl₁).

7. Imaging telesurveillance system (1) according to claim 5, characterized in that said control element (200) comprises means (205,206,210,211) for obtaining a non-linear merging of at least two images of said plurality of panoramic images (LV₁, IR₁, MM₁, Wl₁, W20.

8. Imaging telesurveillance system (1) according to any of claims 5 to 7, characterized in that said control element (200) comprises means (205,206) for enabling or disabling the merging of each of said images of said plurality of images (LV₁, IR₁, MM₁, Wl₁, W2_t) into said global image (101).

9. Imaging telesurveillance system (1) according to any of the preceding claims, characterized in that said means (10, 20, 30, 40, 50) for obtaining a plurality of panoramic images (LVj, IRj, MMj, WIj, W2j) of said area comprise a phase (10) of formation of a visible-light panoramic image (LV;), a phase (20) of formation of an infrared panoramic image (IRi), a phase (30) of formation of a radar panoramic image (MMj), a phase (40) of formation of a panoramic image (WIj) of a reference scene of the area, and a phase (50) of formation of a panoramic image (W2j) containing operational symbols.

10. Imaging telesurveillance system (1) according to claim 9, characterized in that said phase (10) of formation of a visible-light panoramic image (LVj) comprises a plurality of visible-light video cameras (12) adapted to shoot images (12a) of said area, geometric correction modules (14) for correcting optical and geometric distortion on said images (12a) of the area in order to obtain geometrically corrected images (14a), radiometric correction modules (16) for correcting said geometrically corrected images (14a) in order to obtain geometrically and radiometrically corrected images (16a), and a visible-light panoramic image (LVj) composition module (18) for cutting and pasting said geometrically and radiometrically corrected images (16a) so as to obtain said visible-light panoramic image (LVj).

11. Imaging telesurveillance system (1) according to claim 9, characterized in that said phase (20) of formation of an infrared panoramic image (IR;) comprises a plurality of infrared video cameras (22) adapted to shoot images (22a) of the area, geometric correction modules (24) for correcting optical and geometric distortion on said images (22a) of the area in order to obtain geometrically corrected images (24a), radiometric correction modules (26) for correcting said geometrically corrected images (24a) in order to obtain geometrically and radiometrically corrected images (26a), and an infrared panoramic image (IRj) composition module (28) for cutting and pasting said geometrically and radiometrically corrected images (26a) so as to obtain said infrared panoramic image (IR;).

12. Imaging telesurveillance system (1) according to claim 9, characterized in that said phase (30) of formation of a radar panoramic image (MMj) comprises a plurality of radars (32) adapted to shoot images (32a) of the area, and a "ray tracing" module (34) for creating a matching map between each cell of said plurality of radars (32) and one pixel of said radar panoramic image (MMj) by using respective video cameras (35).

13. Imaging telesurveillance system (1) according to claim 9, characterized in that said phase (40) of formation of a panoramic image (Wl₁) of a reference scene of the area generates at least one georeferenced image of the contours of the objects (41,42) located in said area.

14. Imaging telesurveillance system (1) according to claim 9, characterized in that said phase (50) of formation of an image (W2^ containing operational symbols is obtained by analyzing GPS type signals coming from GPS type transmitters placed on objects and/or people located within said area, as well as from multilateration or IFF type systems.

15. Imaging telesurveillance method for monitoring an area, in particular an airport area, comprising the step of: a) obtaining a plurality of panoramic images (LV₁, IR₁₅ MM₁, Wl₁, W2^ of said area, characterized by comprising the additional step of: b) merging together at least two images of said plurality of panoramic images (LV₁, IR₁, MMi, Wl₁, W2j) into a global image (101) providing a larger view of said area, wherein said global image (101) can be represented visually on a display device (300).

16. Imaging telesurveillance method according to claim 15, characterized in that said panoramic images (LV₁, IR₁, MM₁, Wl₁, W2^ are obtained in different frequency bands.

17. Imaging telesurveillance method according to claim 16, characterized in that said frequency bands comprise the frequency band of visible light, the frequency band of infrared light, the frequency band of a radar, and the frequency band of the representation of GPS systems, multilateration systems or IFF type systems.

18. Imaging telesurveillance method according to claim 17, characterized in that said frequency band of a radar is the frequency band of a millimeter radar.

19. Imaging telesurveillance method according to claim 15, characterized in that it also comprises a step for selecting said at least two images of said plurality of panoramic images (LV₁, IR₁, MM₁, Wl₁, W2,) through a control element (200).

20. Imaging telesurveillance method according to claim 19, characterized in that said selection step comprises a step for performing a linear merging of said at least two images of said plurality of panoramic images (LV₁, IR₁, MM₁, Wl₁, W2,).

21. Imaging telesurveillance method according to claim 19, characterized in that said selection step comprises a step for performing a non-linear merging of said at least two images of said plurality of panoramic images (LV₁, IR₁, MM₁, WIi, W2[).

22. Telesurveillance method according to any of claims 18 to 21, characterized by also comprising a step for enabling/disabling the merging of each of said images of said plurality of panoramic images (LV,, IR;, MM;, WIj, W2j) into said global image (101).

23. Imaging telesurveillance system according to any of claims 15 to 22, characterized in that said step a) comprises:

- a step of forming a visible-light panoramic image (LV;);

- a step of forming an infrared panoramic image (IRi),

- a step of forming a radar panoramic image (MM;),

- a step of forming a panoramic image (W 1,) of a reference scene of the area to be supervised;

- a step of forming a panoramic image (W2j) containing operational symbols.

24. Imaging telesurveillance method according to any of the preceding claims, characterized in that said step b) comprises the steps of:

- mapping said plurality of panoramic images (LV;, IRj, MMj, WIj, W2j) to an RGB theme; - fading the visible-light panoramic image (LVj);

- fading the infrared panoramic image (IR;) and the radar panoramic image (MMi);

- deviating the visible-light panoramic image (LVj) chromatically;

- merging together the visible-light panoramic image (LVj), the infrared panoramic image (IRj) and the radar panoramic image (MMi); - overlaying the panoramic image (WIj) of a reference scene of the area to be supervised and/or the panoramic image (W2j) containing operational symbols;

- optimizing the global image (101).

25. Imaging telesurveillance method (1) according to claim 23, characterized in that said step of forming a visible-light panoramic image (LVi) comprises: - a step of shooting images (12a) of said area;

- a geometric correction step for correcting optical and geometric distortion on said images (12a) of the area in order to obtain geometrically corrected images (14a),

- a radiometric correction step for correcting said geometrically corrected images (14a) in order to obtain geometrically and radiometrically corrected images (16a), - a visible-light panoramic image (LVj) composition step (18) for cutting and pasting together said geometrically (14a) and radiometrically (16a) corrected images so as to obtain said visible-light panoramic image (LVj).

26. Imaging telesurveillance method (1) according to claim 23, characterized in that said phase (20) of formation of an infrared panoramic image (IRj) comprises:

- a step of shooting images (22a) of the area to be supervised; - a geometric correction step for correcting optical and geometric distortion on said images (22a) of the area in order to obtain geometrically corrected images (24a);

- a radiometric correction step (26) for correcting said geometrically corrected images (24a) in order to obtain geometrically and radiometrically corrected images (26a);

- an infrared panoramic image (IRj) composition step (28) for cutting and pasting together said geometrically and radiometrically corrected images (26a) so as to obtain said infrared panoramic image (IRi).

27. Imaging telesurveillance method according to claim 25 or 26, characterized in that said geometric correction step for correcting optical and geometric distortion on said images (12a,22a) of the area in order to obtain geometrically corrected images (14a,24a) comprises the steps of:

- defining a distortion model;

- defining a target for the optical tests based on which said distortion model can be estimated, said target being a grid equally spaced in the horizontal and vertical directions;

- defining a distortion model estimation algorithm; - defining a compensation algorithm capable of returning a non-distorted image by applying the inverse model of said distortion model to the acquired image.

28. Imaging telesurveillance method according to claim 27, characterized in that said distortion model is a non-parametric one, allowing to define the shift of each characteristic point of the target, or vertex, caused by distortion.

29. Imaging telesurveillance method according to claim 27 or 28, characterized in that said distortion model consists of a matrix (Δ) containing vectors of shift with respect to the shot image (12a).

30. Imaging telesurveillance method according to any of claims 27 to 29, characterized in that said distortion model is calculated by starting from the luminance of said shot image (12a).

31. Imaging telesurveillance method according to claim 29 or 30, characterized in that said matrix (Δ) is obtained by applying a transformation filter, which implements a Gauss- Laguerre Transform (GLT), and an up-sampling process to said distorted image (12a) for the purpose of obtaining a transformed image.

32. Imaging telesurveillance method according to claim 31, characterized in that the size of said filter is 32, the radial order of the filter is 0, the angular order of the filter is 4, and the scale factor in pixels of the Gaussian function generating the base of the Gauss-Laguerre Transform (GLT) is 13.

33. Imaging telesurveillance method according to claim 31 or 32, characterized in that said up-sampling process up-samples the distorted image (12a) by a variable factor between 4 and 6.

34. Imaging telesurveillance method according to claim 31, characterized in that it additionally includes: a step of finding the local maximum points of said transformed image in order to locate the vertexes of said transformed image; a step of matching the vertexes of said transformed image to the vertexes of the non-distorted target so as to order the elements of said matrix (Δ), said elements containing the coordinates of the vertexes in the form of fractional numbers.

35. Imaging telesurveillance method according to claim 25 or 26, characterized in that said radiometric correction step for correcting said geometrically corrected images (14a,24a) in order to obtain geometrically and radiometrically corrected images (16a,26a) comprises a step of defining a piecewise function by which an image on the left and an image on the right of a pasting line are multiplied, said function being characterized by:

- assuming the unit value on the central vertical line of each image, thus not altering it;

- leveling out the values of the pixels of the two pasted images on the right and on the left of the pasting line, thus bringing them to the mean value of the same pixels in the original images;

- being linear in the horizontal direction.

36. Imaging telesurveillance method according to claim 23, characterized in that said step of forming a radar panoramic image (MMj) comprises:

- an off-line calibration step, wherein a matching map is created between each range cell of a radar (32) and one pixel of said panoramic image (MMj) by defining ground points and a

"ray tracing" step on a focal plane (PF); - a real-time step.

37. Imaging telesurveillance method according to claim 36, characterized in that said offline calibration step comprises a georeferencing step, a step of finding said ground points, and a step of projecting said ground points on said focal plane (PF).

38. Imaging telesurveillance method according to claim 23, characterized in that said step of forming a panoramic image (Wl;) of a reference scene of the area generates a georeferenced image of the contours of the objects (41,42) located in said area.

39. Imaging telesurveillance method according to claim 23, characterized in that said step of forming an image (W2j) containing operational symbols is obtained by analyzing signals, preferably GPS type, coming from transmitters, preferably GPS type, placed on objects and/or people located within said area, as well as from multilateration systems or IFF type systems.

40. Information technology product which can be loaded in the memory of at least one computer and which comprises software code portions for implementing the method according to any of claims 15 to 39.