WO2000052643A1

WO2000052643A1 - Endoscopic observation device

Info

Publication number: WO2000052643A1
Application number: PCT/CH2000/000096
Authority: WO
Inventors: Charles Baur; Gaëtan MARTI; Nicolas Chauvin
Original assignee: 2C3D Sa
Priority date: 1999-02-26
Filing date: 2000-02-22
Publication date: 2000-09-08
Also published as: EP1155383A1; FR2790196A1

Abstract

The invention concerns a device for observing a field (10) comprising: two video cameras (18a, 18b) for capturing a common area of the field, an electronic device (20) for processing signals supplied by the cameras, and communication means (22) for providing an operator, on the basis of the signals supplied by the electronic processing device, data concerning the observed field. Said electronic device (20) comprises: monitoring means for selecting and identifying two points common to the images acquired by the cameras, means for determining from the data supplied by the monitoring means, a value representing the distance separating said two points in a three-dimensional space, and processing means for transforming said value into signals which are supplied to the communication means to enable them to provide the operator with the data concerning the observed field.

Description

ENDOSCOPIC OBSERVATION DEVICE

The present invention relates to an endoscopic observation device of a three-dimensional (3D) field intended, in particular, and not exclusively for surgery.

The object of the present invention is to allow the determination at high video frequency (that is to say at more than 25 Hz) of the distance separating a surgical tool from the various elements constituting the operating area and thus to provide the surgeon with the spatial data necessary for the proper execution of operational gestures.

The endoscopic approach is a surgical technique which becomes more and more important every day. The principle of this approach, called "minimally invasive" approach, consists in making short incisions through which, via an adapted device, the tools necessary for the intervention are penetrated into the patient's body as well as a camera or endoscope. providing on a screen the images which allow the vision of the operating field. This technique has such a potential to reduce operational trauma and, at the same time, the costs associated with hospital stay that all sectors of surgery are interested in it: digestive surgery, gynecology, ENT, cardiology, bone surgery, ...

One of the biggest problems related to endoscopic techniques is that the working environment is visualized in two dimensions (2D) while the operating gestures are performed in three-dimensional space (3D). Several attempts have been made, to date, to capture and provide in real time (25 Hz) to the surgeon the three-dimensional information necessary for the execution of a surgical procedure. A first known approach consists in using mechanical arms at the end of which the tools are fixed. It is penalized by too much restriction of the authorized movements and a large footprint of the operating area, especially when several tools are used simultaneously.

A second approach is optical monitoring, which consists of attaching active elements (LEDs) monitored by cameras to the instruments. However, the presence of cables, angular inflexibility and sterilization problems made this method little known.

A third approach is passive monitoring, which consists of mounting benchmarks (generally three spheres) on the tools that a vision system pursues. Congestion problems are certainly reduced, but they remain. Furthermore, it is not possible to completely eliminate the risk of losing position information due to an obstruction of the visual field "cameras - markers" following, for example, a gesture by the interveners in the operating area.

Inspired by the development of interfaces used in virtual reality to be able to visualize computer graphics environments in 3D, new approaches have been proposed. Using different types of glasses (passive or active), they exploit the ability of the human brain to reconstruct the concept of depth by presenting images with a parallax for the right eye different from that for the left eye. Problems of discomfort associated with the wearing of glasses, of potential interruption of the synchronization signal in the case of active glasses and, above all, the restitution of a subjective image have, in this case also, limited the diffusion of such approaches.

Another similar device is described in US Pat. No. 4,935,810. It includes two cameras and two screens on which the respective images of the cameras appear. By selecting particular points on the screens, it is possible to calculate the distance between them. Such a solution makes it possible to know, in an operating field, the distance which separates an object, for example a scalpel, from an organ to be operated. Unfortunately, like the operations of shooting, selection of the points considered and calculation of the distances is done sequentially, it is not possible to continuously monitor the operating field. This results in significant risks of handling errors.

The object of the present invention is to provide an endoscopic observation device free from the drawbacks and limitations of the devices currently known.

More specifically, the device according to the invention is characterized in that it comprises:

- at least two video cameras (18a, 18b) each associated with an optic making it possible to apprehend a common space of the field to be observed and delivering an electrical signal representative of the captured images,

- electronics (20) for processing the signals supplied by the two devices, - means for storing representations of objects likely to appear in said fields,

tracking means for automatically identifying at least two points P1 and P2 common to the captured images, at least one being associated with one of said representations, and for producing information relating to the position of these points in a space to three dimensions,

- calculating means (26c) for determining, from said information, a value representative of the distance separating said points, - processing means (32) for transforming said representative value into signals, and

- communication means (22) for supplying an operator, on the basis of signals from the processing means, with information relating to this distance. Preferably, the two optics are rigidly associated with each other to form a stereoscopic endoscope. Besides, they have axes parallel, spaced from each other by a distance D, equal focal distances f and coplanar focal planes. The information produced by the monitoring means is then, for each of the two points, constituted by the coordinates xL, xR and y of its image in the focal plane of the corresponding optics, xL being the abscissa of the image of the point in the left space, xR the abscissa of the image of the point in the right space and y the ordinate of the image of the point in the left and right spaces.

The calculation means are advantageously arranged to carry out the operations of: - calculation of the coordinates Xp ,, Yp, and Zp, of the point P, and Xp ₂ , Yp ₂ and

Zp ₂ of point P ₂ in a cyclopean space according to the formulas: X = (D / Δ). (xL + xR) Y = D. y / Δ Z = D. f / Δ with Δ = xL - xR

- calculation of differences:

Yp, - Yp ₂

- determination, from these three differences, of the distance separating the two points in a three-dimensional space by the formula: d ₁₂ = [(X _P1 - Xp ₂ ) ² + (Y ₁ - Yp ₂ ) ² + ( Z _P1 - Zp ₂ ) ² ] ¹ ' ²

According to a preferred embodiment, the device according to the invention also has the following characteristics:

the means of communication include a video screen making it possible to display an image of the field observed;

the processing means are arranged to generate a significant image of said representative value and superimpose it on the video screen on an image of the field observed; - this signifying image can be, among many possible solutions, an index or an area having a color gradient;

- it also includes:. means for synchronizing the images of the two cameras,

. an analog-digital converter interposed between each camera and the processing electronics,

. a memory for storing the signals from the converters,. selection means, forming part of the processing electronics, so as to take into consideration only a part of the information coming from at least one of the converters, in order to reduce the volume of information processed,

. correction means for processing the images so as to reduce the effect of camera aberrations, and

. a switch for limiting the information displayed on the screen to the image from one of the cameras.

Other characteristics of the invention will emerge from the description which follows, made with reference to the appended drawing, in which: - Figure 1 illustrates, diagrammatically, an assembly comprising a device according to the invention applied to a surgical operation;

- Figure 2 shows the general structure of the device according to the invention;

- Figure 3 shows, in more detail, part of the structure of Figure 2;

- Figure 4 illustrates the principle of calculating the distance between two points in a three-dimensional space;

FIGS. 5a, 6a, 7a and 8a represent class diagrams, respectively the main diagram of the application and the diagrams relating to the acquisition, the stereo and the monitoring of tools, while FIGS. 5b, 6b, 7b and 8b are diagrams of sequences corresponding to class diagrams with the same number.

Figure 1 shows, very schematically, the means used, according to the invention, for a surgical operation using an endoscopic observation device. The operation takes place in an operating field 10 internal to an organism and allows intervention on an organ 11 by means of a tool 12 introduced into the organism through an incision 14a. The surface of the member 11 is provided with marks 11a consisting, for example, of dots produced by means of a biocompatible ink, adherent pads, or also of certain parts of the member itself which have a particular appearance. . Marks 12a, generally formed by colored dots, are advantageously placed on the tool 12, so as to facilitate its identification and, as will be explained below, to calculate its position. In the device according to the invention, the observation of the operating field 10 is done by means of a conventional endoscope 16 with double optics, associated with a light source (not shown) and introduced into the organism by another incision 14b . The endoscope 16 is connected to an image capture device 18 comprising two cameras 18a and 18b, called respectively the left camera L and the right camera R, which receive the light radiation captured by the two optics, so as to be able to process the images stereoscopically, as will be explained later.

The cameras 18a and 18b transform, in a conventional manner, the light radiation coming from the operating field 10 into an electrical signal which is applied, by two distinct channels L and R, to a processing electronics 20. A video screen 22, taking the place of communication device, displays at a frequency of 25 Hz the images received by the cameras 18a and 18b, before or after processing by the electronics 20. A switch 23 makes it possible to select one or the other of the cameras 18a and 18b or another image obtained after processing by the electronics 20. As described below, the device according to the invention makes it possible to define the distance between two marks 11a and / or 12a visible in the operating field 10, but also between all points identifiable by their shape, their color, etc. Reference will now be made to FIG. 2 which represents the general structure of the device according to the invention. There is the operating field 10 with the organ to be treated 11, the intervention tool 12, the endoscope 16, the image capture device 18, as well as various modules constituting the processing electronics 20. The field 10 is attached to a Cartesian frame of reference whose Z axis is parallel to the axes of the cameras.

The image capture device 18 comprises, in addition to the two left cameras 18a and right 18b, an image synchronization circuit 18c and two analog-digital converters 18d and 18e.

The synchronization circuit 18c makes it possible to manage the images coming from the two cameras so that they are perfectly synchronized, which facilitates their comparison, as will be explained below, and improves the quality of the information relating to the third dimension.

The two analog-digital converters 18d and 18e transform the signals coming from the synchronization circuit 18c, of analog type, into signals of digital type.

The output of the device 18 is connected to the input of a digital video recorder 24 which can record all or part of the operation. In a variant not shown in the drawing, it is also possible to use an analog recorder which is then connected to the input of the converters 18d and 18e.

The output of the device 18 is also applied to the processing electronics 20 advantageously constituted by a computer in which the different modules are defined by a set of programs and subroutines described with reference to FIGS. 5 to 8. This computer comprises conventional control means, not shown in the drawing, such as a keyboard and / or a mouse.

As a variant, the electronics 20 could be produced by means of different electronic modules. This second solution, however, offers less flexibility of use.

In order to facilitate understanding, the device according to the invention will be described, with reference to Figures 2 and 3, on the basis of a modular construction in which each module performs a particular function. Figures 5 to 8 will then allow a better understanding of the structure of the software controlling the device.

The processing electronics 20 is arranged so as to process the signals from the two cameras in parallel. However, to lighten the design, each of the modules or systems is shown only once.

The heart of the electronics 20 is constituted by a signal processing module 26, which includes systems for tracking objects 26a and 26b, intended to follow the possible movements of the objects (tools and organs) to which the marks 1 1 a and 12a are linked, as well as a unit 26c for calculating the distance separating, in a three-dimensional space, at least two points associated with these marks. The latter are selected, for example, by means of the computer mouse, from among a set of representation of objects, stored, likely to appear in the operating field. They are identified by the object tracking systems 26a and 26b.

The module 26 alone makes it possible to determine the position in the space of points of interest identified by the operator using the marks 11 a and 12a, as well as the distance separating them. However, in order to give the device greater flexibility of use and optimal precision, the electronics 20 further comprises:

- an image acquisition module 28 comprising an interface 28a and a memory 28b; a correction module 30 comprising a filtering circuit 30a of the signals coming from at least one of the converters 18d and 18e, with a view to selecting a part thereof, and a circuit 30b for correcting the aberrations of the cameras; and a screen control module 32, which includes a system for combining images 32a and a screen control 32b.

The acquisition module 28 is connected, by its input, to the output of the image capture device 18. It makes it possible to store information and can thus play a role of black box, recording in its memory 28b, the content of which cannot be altered, all or part of the information relating to the progress of the operation. Its interface 28a makes it possible to transform the information received to put it in a form compatible with the characteristics of the memory 28b.

The correction module 30 is connected, by its input, to the output of the acquisition module 28. Its filtering circuit 30a makes it possible to process the images to make them more readable. By applying an adequate algorithm to them, we can only keep the outline of the objects present in field 10 or, again, represent only one of the colors of the images, reinforce the contrast, etc. This allows you to have a different perspective on the operating field and thus better understand certain details. As for the correction circuit 30b, its function is to correct the aberrations of the optics of the endoscope. This correction is essential in order to have a good image. Indeed, the optics of endoscopes are very small. This results in a strong distortion of the images. To overcome this drawback, a plane transformation is established which makes it possible to find correspondences between the points of the ideal image and of the distorted image. This method is described in detail in "Digital Image Warping, George Wolberg, IEEE Computer Society Press Monograph, 1994".

The signals available at the output of the module 30 thus have characteristics making it possible to display, on the video screen 22, information which is easier for the surgeon to interpret. These signals are introduced into the processing module 26 which will be described in more detail with reference to FIG. 3.

Finally, the control module 32 manages the information displayed by the video screen 22, whether or not combining the images of the operating field 10 with information concerning the position of the various objects present in the field 10. The system for combining images 32a, called "multimage", by contraction of the words "multiple" and "image", or, in English, "overlay". It has its input connected to the output of module 26 and processes the signals supplied by it at the same time as the signals produced by the camera 18a or 18b. The screen control 32b, connected to the output of the system 32b, is connected by its output to the video screen 22 which makes it possible to view the operating field 10 as well as information relating to the organs and tools, in particular information relating to the distances.

The indication of distances can be done in different ways. The value of the Z coordinate of the distance between the tool 12 and a selected mark 11a can be displayed in digital form or, for example, using two indexes in V. It is also possible to represent the distance between a given point, for example the end of the tool 12, and all or part of the field 10 by a gradient of colors, ie blue corresponding, for example, to very distant areas and red on contact.

As soon as the tool 12 must approach with care a particular point, this point being identified, it is also possible to show a part of the tool 12 in a color corresponding to the measured distance.

The choice of one or the other solution, achievable at all times, will be made according to the problems with which the surgeon is confronted, giving more or less importance to information relating to the value of the Z component, or on the contrary in the operating field.

It is also possible to add additional images, coming from a database not shown in the drawing and facilitating a diagnosis, for example reticles, masks or any other image. likely to facilitate the surgeon's work. The procedure for obtaining this superimposition of images will be specified with reference to FIG. 5.

To determine the distance between two marks belonging to objects arranged in the field 10, it is necessary to establish a correlation between the different objects visible on the two images captured by the cameras 18a and 18b. As the shots are taken from two distinct points, it follows that the distances between the projections of two brands on the plans of the cameras, transposed into a common reference frame, are different. By calculating the difference of the coordinates of these projections in the common reference frame, it is possible to calculate the distance separating these marks in three-dimensional space.

Figure 3 provides a better understanding of how to do this. It represents the detail of the calculation unit 26c which comprises, connected in series, a geometric correction entity 260 intended to process the images in epipolar geometry, a contour detection entity 262, a correlation entity 264, a determining the distance 266 and a filter 268.

The geometric correction entity 260 makes it possible to process the images in epipolar geometry. To fully understand this geometry, we will advantageously refer to the article by Zhengyou Zhang entitled "Determining the Epipolar Geometry and its Uncertainty: A Review" and published in Journal of Computer Vision, 1998.

The contour detection entity 262 makes it possible to select, identify and follow the marks 11a and 12a, as well as particular areas of the objects located in the field 10. The device can thus recognize the various objects present in the field and follow them in successive images. The means used to carry out this recognition are fully described in the publication by Kurt Konolige entitled "Small Vision: Hardware Implementation" and published in the annals of "Eight International Symposium on Robotics Research, Hayama, Japan 1997". From the information obtained by the contour detection entity 262, it is possible to examine two same points on the two images, to define the distance which separates them on one and the other of these images and to make the difference between these two distances. This difference, called disparity, is defined by the correlation entity 264.

By knowing the characteristics of the optics and the disparity, it is possible to establish a relationship between the disparity and the distance between the two points. This operation is performed in the distance determination entity 266 and will be described in more detail with reference to FIG. 4. Finally, the filter 268 ensures the elimination of inaccuracies, for example by applying a spatial interpolation method and temporal, such as that defined in the 2013 INRIA research report (1993) entitled "Real-time correlation-based stereo: algorithm, implementations and applications", established by Olivier Faugeras. FIG. 4 makes it possible to understand the manner of determining the coordinates of two points common to the images captured by the cameras, then of calculating the distance which separates them. This figure shows two optics L (left) and R (right), with the same focal length f, whose optical axes are parallel to each other and distant by a value D. In addition, their focal planes are coplanar. We also see a point P belonging to field 10 and whose position it is to determine.

For this purpose, there are two plane referentials, one left and the other right, whose respective abscissae, aligned on the same axis, are xL and xR and whose ordinate is y. The distance between the origins of these reference frames is equal to D. They respectively make it possible to define the position pL of the image of the point P seen by the left camera and its position pR seen by the right camera. It is thus possible to measure the coordinate of the images of point P in the left and right repositories and thus define its xLp, xRp and yp coordinates. It will be noted that the determination of the coordinates in the left and right frames of reference is done simply by identifying the pixels of the cameras on which the image of the points considered is formed. The units of xL, xR and y are therefore pixels. The focal length f must also be expressed in this unit.

To define the position of point P in the field, we use a third frame of reference whose axes X, Y and Z define a space called cyclopean. The X-Y plane is parallel to the xL / xR-y plane located at the front, at a distance equal to f. The Z axis is parallel to the optical axes and arranged in the same plane, in the middle position.

To calculate the coordinates of point P in cyclopean space, we first define the disparity Δ of its images according to the formula: Δ = xL - xR

We can then define the coordinates of point P in cyclopean space according to the formulas:

Xp = (D / 2Δ). (xL + xR) Yp = D. y / Δ Zp = D. f / Δ

When it is a question of calculating the distance between two points P- and P ₂ , one must thus determine, first of all, according to the method above, the coordinates Xp. ,, Yp., And Zp, of the point P, and the coordinates Xp ₂ , Yp ₂ and Zp ₂ of the point P ₂ . The distance d ₁₂ which separates them is then obtained by the formula: d ₁₂ = [(Xp, - Xp ₂ ) ² + (Y _l - Yp ₂ ) ² + (Z _P1 - Zp ₂ ) ² ] ¹ ' ²

If, in the above formulas, D and ex _P rhyme in mm, the values of X, Y and Z will also be.

In the device according to the invention which has just been described, the cameras 18a and 18b permanently capture the images of the operating field 10, through the endoscope 16. These images are synchronized by the circuit 18c and converted from the analog mode in digital mode by the 18d and 18e converters. The signals are then sent to the recorder 24 to store the operation and to the image acquisition module 28 which allows, on the one hand, to modify them by means of the interface 28a by playing, for example, on the contrast, the brightness, etc. and, on the other hand, to store them in the tamper-proof memory 28b, which can act as a black box. The signals thus obtained are applied to the correction module 30 which eliminates certain faults affecting the quality of the images. They are then sent to the processing module 26 which calculates the distances and the monitoring of the organs, then to the control module 32 which superimposes the images before controlling the video screen 22. As has been explained above, the essential functions of the device according to the invention can be provided by software, advantageously written in an object-oriented language. This software is schematically represented by means of a class diagram in FIG. 5a and a diagram of the sequences of the main loop in FIG. 5b. These diagrams call for analysis using the UML methodology described in "UML, the unified object modeling notation, application in Java" by Michel Lai, InterEditions 1997, ISBN 2-7296-0659-9 and executable using the "Rational Rose" software offered by Rational Software.

In the following description, a set of signals will be called "image" which, duly processed, make it possible to form an image on a screen. In addition, we will call "multimage" an approach allowing to superimpose images, while the term "monitoring" will designate the part of the program which makes it possible to follow an organ and a tool as long as it is in field 10, in order to be able determine the distance between them. In the diagram in FIG. 5a, the software is structured into classes forming, by a reference link, the application which is itself a class and bears the reference 40. Each class is corrφosed with operations and d 'attributes. On this diagram, we could see an acquisition class 41, which will be described in a _P read _P manner with reference to FIG. 6 and display classes 42, conversion 44, image 46, User interface 48, "multimage" 50, "multimage "relative 52 and" multimage "absolute 54, stereo 56," followed "by organs 58, and" followed " of tools 60. The stereo classes 56 and "tracking" of tools 60 are respectively shown in more detail in FIGS. 7 and 8.

In some of the classes mentioned above, it is possible to adjust, automatically or voluntarily, one or more parameters that comprise the attributes, which gives the device great flexibility of use.

The various attributes and operations contained in the classes are formulated in Java language, in the UML notation mentioned above.

The acquisition class 41 contains the operation in _P elée "new images (right: Image, left: Image)" which makes it possible to process the images coming from the cameras 18a and 18b. This operation acquires the left and right images synchronously, corrects them and stores them. It will be described more precisely with reference to FIG. 6.

The display class 42 contains the "display (input: Image)" operation which allows the display screen to be controlled from the signals coming from the different classes making up the application.

The conversion class 44 contains the operations:

- "RGBenYδ (input: Image, output: Image)" which allows you to convert a color image in RGB format to black and white image in Y8 format, - "RGBenHLS (input: Image, output: Image)" which allows you to convert RGB image to HLS (Hue, Luminance, Saturation) image, and

- "reduce (factor: float, input: Image, output: Image)" whose mentioned factor defines a reduction rate of the resolution of an image.

The image class 46 contains the information relating to an image. Its attributes are:

- "format: Formatlmage", which contains the information of the internal format of the image, that is:

- Y8 format (shades of gray)

- RGB format (16, 24 or 36 bit color), and - HLS format (36 bit color) - "width: int", which defines the width of the image,

- "height: int", which defines the height of the image, and

- "table: Byte ^* " which acts as a pointer to the pixels of the image.

Concerning the RGB and HLS formats, we will advantageously refer to the publication by Ken Fishkin & San Rafaël entitled "A fast HLS-To-RGB Transform", published in Graphics GEMS, 1990, pages 448-449.

The user interface class 48 contains the "modifyParameters (newParams: SystemParameters)" operation which manages the interaction between the user and the various system input devices (keyboard, mouse, voice recognition, etc.). This operation modifies, on the operator's order, the parameters of the stereo, organ monitoring, in particular the identification of points to follow, tool monitoring and absolute and relative "multimages", which will be described later. with reference to classes 50, 52, 54, 58 and 60. The "multimage" class 50 contains the "calculate (tools: ListTools, brands: ListBrands, image entry: Image, Stereo input: Image, output: Image) operation. Two subclasses, called "relative multimage" 52 and "absolute multimage" 54, are derived from the class "multimage" 50.

Class 50 allows the calculation of the superposition of images in absolute or relative mode according to selected parameters, as will be explained below, and more particularly to ensure the mixing of 2D type images, corresponding to a usual vision of the field 10 and to add representative images of the third dimension. In the case of relative "multimage", the distances are defined between two marks present in the field 10, while in absolute "multimage", the distances are defined relative to the endoscope.

The relative "multimage" subclass 52 contains the "calculate (tools:

ToolsList, brands: BrandsList, image input: Image, Stereo input:

Image, output: Image) "which calculates the superposition of the images in relative mode according to the related parameters. Its attributes "ParametersMultimageRel" parameters which are the choice of the type and the variables relating to this type. With this class, it is possible to superimpose additional information on the image of the field by creating one or more hollow and virtual spheres centered on the end of the tools, for example, and by representing the geometric location of their intersections with the or the organs visible in the field 10 by a modification of the tone and / or the saturation of the corresponding pixels on the image. The end of the tool can also be provided with a virtual light oriented in the extension thereof, which modifies the brightness, the tone or the saturation of the field 10. It is also possible to display the distance between a tool and a mark defined by the user by means of digital information, cursors or any other means, such as a color gradient.

The absolute "multimage" class 54 contains the "calculate (tools: List of Tools, brands: List of Brands, image input: Image, Stereo input: Image, output: Image)" operation which allows to calculate the superposition of images in absolute mode as defined. above. Its attributes are the "ParametersMultimageAbs" parameters which are the position and resolution of the "multimage" as well as the color gradient.

The stereo class 56 will be described in more detail with reference to FIG. 7. It can however already be specified that it contains the operations of:

- "calculate (input Right: Image, input Left: Image, output: Image)" which allows the calculation of the stereo image, also called "range image", according to the parameters below,

- "merge (small Input: Image, large Input: Image, output: image)" which allows the fusion of two disparity images not having the same resolution.

Class 56 has for attributes the "StereoParameters" parameters which are the following:

- Parameter of the system geometry for the conversion of the disparity image into a distance image, - Parameters for the correction of the epipolar geometry,

- Offsets to correct the respective position of the two images, - Search for disparities,

- Size of search windows for pattern recognition,

- Confidence threshold,

- Parameters for multi-resolution (respective position and resolution of the different images), and

- Parameters for post-filtering.

In the present description, a “shape recognition” method is used to establish correspondences between the equivalent points of the two images, left and right. This method is described in the publication of Kurt Konolige already cited.

The "follow-up" class of organs 58 contains the operations "addBrandAfollow (entry: Image, brand: Position2D)" and "followBrand (entry: Image): listBrand", which allow the user to add marks 11a to follow . Its attribute is the "ParametersSuriviOrg" parameters which are as follows:

- Number of brands to follow,

- Form of marks to follow,

- Previous position of the brands to follow, and

- Current position of the brands to follow. The "tracking" class of tools 60 will be described in more detail with reference to FIG. 8. It can however already be noted that it contains the operation: "search (entry: Image): listTools" having the function of searching the tools in the picture. Its attribute is the parameters: "ParametersTrackingOut" which are the following: - Maximum number of tools in the field, and

- Types of tools possible.

In the software, the interaction of the instances of classes defined above is implemented by the application of its operations, according to a sequential procedure re _P represented on the logic diagram of FIG. 5b. Classes are re résentées _P x-axis and have the same references as in Figure 5a, and their name. Operations to perform the application are represented on the ordinate, identified by the name of the parameters enabling them to be written written above an arrow connecting the application 40 to the class containing this parameter.

The sequence shown begins with the acquisition of a new image, defined by the operation "nouvelleslmages (Image, Image)" contained in the acquisition class 41. The operation "RGBenYδ (Image, Image)" converts the images received in Y8 format, then the "reduce (float, Image, Image)" operation allows you to choose the resolution of the image. These two formatting operations with a view to subsequent processing are contained in the conversion class 44. The stereo class 56 makes it possible, by the "calculate (Image, Image, Image)" operation, to calculate a depth image and, by there, the distances between the different marks present in field 10.

The two following operations, "followBrand (Image)" and "search (Image)", contained respectively in the classes "follow-up" of organs 58 and "follow-up" of tools 60, allow the follow-up of marks in the field, associated respectively to the organs and tools therein.

The operations "RGBenHLS (Image, Image)" and "calculate (List of Tools, List of Brands, Image, Image, Image)", contained respectively in the conversion classes 44 and "multimage" 50, allow to create an image which, superimposed on the image of the field 10, provides information on sé distances _P Arant brands associated with organs and tools.

Finally, IO _P eration "display (Image)", contained in the display class 42, transmits on the screen the information _P allowing to see the cham _P 10 embraced _P by the cameras 18a and 18b, as well as the information relating to the distances .

As shown in Figure 6a, the 41 acquisition class itself contains the classes _PP elées correcting o tick 411, filter 412, storage 413, Eri _P _P hérique acquisition 414 and synchronizer 415. The _P Eri _P hérique d acquisition 414 is linked to acquisition class 41 through class 415. The optical correction class 411 contains the "correct (input: Image, output: Image)" operation which enables errors due to the optics of the endoscope to be corrected. Its attribute is the "CorrectionParameters" parameters. Through this class, the information coming from the cameras 18a and 18b is processed so that their coordinates in x and y are corrected according to the optical characteristics of the endoscopes, on the basis of a distortion correction algorithm o _P ticks described in the publication entitled "Digital Image Processing (second edition)", IEEE Com _P uter Society Press Monograph, 1994, by Wiliam K. Pratt. The filtering class 412 contains the operation "filter (input: Image, output: Image)". Its attributes are the "FilterParameters" parameters. This operation processes the images from the cameras 18a and 18b, eliminates the parasites linked to the even and odd lines and corrects the light intensity. The image obtained, after the operations of these two classes have been applied, is therefore of suitable quality for further processing.

The storage class 413 contains the operation "save (input: Image)" which records all or part of the images on a digital medium, in their state before and / or after processing by the operations of classes 411 and 412. It is also possible memorize the different states of the system, the successive positions of the tools and the interactions with the system of different partners, notably the surgeon. This memorization makes it possible to preserve the images in the event of a problem. Advantageously, the memory intended to receive them is of permanent type, so that its content cannot be modified and it can thus take the place of proof. The acquisition peripheral class 414 contains the "acquire (): Image" operation. Its attributes are the "DeviceParameters" parameters which are as follows:

- type of signal,

- image format, and - image resolution and position. The video signal, initially of analog type, is transformed into digital mode _P by the operation contained in this class.

The synchronizer 415 class contains the operation "acquérir2lmages (Left: Image right: Image)" which guarantees a synchronous acquisition of left and right image obtained has _lmost treatment _{I n} instances of the _P Eri hérique acquisition 414 and delivers the images thus acquired to the acquisition class 41.

The pattern sequences of the data acquisition is re _{BSB has} to Figure 6b, according to the same Rincí e as those applied in Figure 5b. It defines the detail of the sequence which takes place entirely around the acquisition class 41. It will be noted that the images coming respectively from the cameras on the right and on the left are acquired simultaneously, but processed successively. This detail is not shown in this figure, however.

After the "newslmages (lmage, Image)" operation has been performed, the instances of the acquisition class 41 carry out the operation "acquire2lmages (lmage, Image)" and give the orders to the instances of the synchronizer class 415 of perform the "acquire ()" operation, for both the right and the left cameras, and this in synchronism. These images are then put into memory by the "save (image)" operation contained in the storage class 413, then their geometry is corrected by the "correct (image, image)" operation. They are finally filtered by the "filter (image, Image)" operation. The program then returns to the main loop of Figure 5b to perform the "RGBenY8 (image, Image)" operation.

FIG. 7a shows the classes derived from the stereo class 56, and more particularly the epiσolar correction classes 561, edge detection 562, correlation 563 and stereo conversion 564.

The epipolar correction class 561 contains the "correct (entry:

Image, output: Image) "which transforms the epipolar lines into parallel lines. Its attributes are the parameters" ParamGéomEpip "necessary to ensure a correction of the images by means of the epipolar geometry, or the transformation matrices specific to a given tick. All the information that _P ro _P bones are found in the book "Determining the E _P i _P olar Geometry and Its Uncertainty: A Review," _M ore cited above.

The edge detection class 562 contains the operation "filterLOG (input: Image, output: Image)" which filters the image by the method known as LOG (La lacian of Gaussian) described in the publication entitled "A computation theory of human stereo vision ", Proceeding Royal Society B-204- 1979, by Marr D. and Poggio T. This class has as attributes the parameters" Filter Parameters "which are the coefficients of the LOG filter. The correlation class 563 contains the operation "correlate (right: Image, left: Image, output: Image)" which makes it possible to define the correspondence between the pixels of the image on the right and those of the image on the left, and build an image of disparity. Its attributes are the "CorrelationParameters" parameters which are as follows: - correction offsets of the respective position of the left and right images which allow these images to be adjusted so as to be able to ensure the correlation between them,

- disparity search space, confined to the only area for which it is useful, thus making it possible to avoid unnecessary calculations, - size of the search windows for "pattern recognition", also making it possible to reduce the volume of information processed, and

- confidence threshold, which offers the possibility of defining a minimum threshold for the quality of distance measurements.

The stereo conversion class 564 contains the "disparityInDistance (input: Image; output: Image)" operation which transforms the disparity into millimeter distance, taking into account the geometry of the system optics. Its attributes are the parameters "ParamGéomOptique" which are the parameters of the geometry of the optics of the device for the conversion of the disparity image into a distance image. FIG. 7b shows the sequence diagram for the calculation of the stereo, structured according to the same principles as for FIGS. 5b and 6b. In this figure, the operations performed on the left and right images have been shown in a specific manner, in order to better differentiate the operations relating to the left and right images from the common operations.

The application accesses stereo class 56 by the order "calculate (image, Image)". The "correct (image, Image)" operation of the epipolar correction class 561 transforms the left image into epipolar coordinates. The "filterLog (Image, Image)" operation of the contour detection class 562 makes it possible to define the contours of the various objects present in the field 10 and visible on the left image. The same operations are then performed on the right image. The operation "correlate (image, Image, Image) of the correlation class 563 then ensures the correlation between different points of the left and right images. The operation" disparityInDistance (Image, Image) "of the stereo conversion class 564 determines the distances from the disparity values The program then continues in the main loop, as shown in Figures 5a and 5b.

Figure 8a shows the complementary classes derived from the tool tracking class 60. We can see the detection classes 601, filtering tracking 602, tools 603 and marks 604.

The detection class 601 performs the operations "detectBrand (input: Image): listBrand" which makes it possible to define the position of marks in the filtered images using information relating to the geometry and colors of tools and marks, and "detectTools ( brands: list brands): list tools ". By this operation, the program determines the different significant points present in the field 10 and representative of a tool. It corrects their alignment and calculates the position and orientation of the tool in space to represent it. It can, moreover, regulate the position of the end of the tool, in order to have a more stable image. The detection class 601 has for attributes the tools "listTools", defined in class 603 and which will be examined later. The filtering class followed 602 performs the operation "filter (input: Image, output: Image)" which allows to keep only the images relating to brands and tools but to erase the background. This filtering is done in HLS format. It also has for attributes the tools "listTools". The tools class 603 contains all the information enabling the different tools present in field 10 to be identified. Its attributes are the following parameters:

- "type: TypeOutil" which contains all the information relating to the tools, such as their geometry and their marking, - "orientation: 3D orientation" relating to the information making it possible to define the orientation of the tool in space,

- "position: Position3D" containing information relating to the position of each of the tools in space, and

- "end: 3D position" used to define the position of the end of the tools in space.

Taking into account the end of the tools in a specific way allows you to work with maximum precision. This is particularly important with scalpels.

The 604 brand class has the following parameters as attributes: - "center: 2D position", giving information relating to the center of gravity of the brands present in field 10,

- "surface: int" which defines the surface of each of the marks,

- "geometry: Geometry" which contains information relating to the shape of each of the marks, and - "color: Color" in which the colors of each of the marks are stored.

If we refer to Figure 8b, we see that we access the class followed by tools 60 by the order "search (Image)". The filtering class followed 602 contains the operation "filter (Image, Image)" which makes it possible to have an image on which there are only the tools and the marks. The detection class 601 contains the operations "detectBrands (image)" and "detectTools (List Notes)", then the program returns to the main loop.

The device according to the invention, as just described, allows surgical operations by endoscopic technique offering a maximum of information that can be called upon request, without requiring equipment hindering the work of the surgeon and of his team. It can also be used in the field of orthopedic surgery.

The same concept is absolutely not limited to the medical field. It applies to many other situations such as inspecting pipes, determining the exact position of fixed or moving objects in a given space that is difficult to access, etc.

It may also be noted that in addition to the indication of information relating to the third dimension, other information may be associated with the displayed image, without however departing from the scope of the invention.

In a variant not shown, the information relating to the third dimension can be transmitted by sound rather than optically, the device modifying, for example, the frequency of a signal transmitted, or even giving the information in clear. In another variant, a cross-linked network is projected in infrared light onto the operating field, invisible to the eye, but which the cameras can detect. Such a network can be used to improve the accuracy of the measurements, the number of marks thus being considerably increased.

Claims

1. Field observation device (10), characterized in that it comprises:

- electronics (20) for processing the signals supplied by the two devices,

- means for storing representations of objects likely to appear in said fields, - monitoring means for automatically identifying at least two points P1 and P2 common to the captured images, at least one being associated with the one of said representations, and to produce information relating to the position of these points in a three-dimensional space, - calculation means (26c) for determining, from said information, a value representative of the distance separating said points,

- processing means (32) for transforming said representative value _P into signals, and - communication means (22) _T o supply to an operator, from the signals coming from the processing means, information relating to this distance.

2. Device according to claim 1, characterized in that the two optics are rigidly associated with each other to form a stereoscopic endoscope (16).

3. Device according to claim 2, characterized in that:

the said optics have parallel axes, spaced from each other by a distance D, equal focal distances f and coplanar focal planes, and the information produced by said sighting means is, for each of said two points P, and P ₂ , constituted by the coordinates xL, xR and y of its image in the focal plane of the corresponding optics, xL being the abscissa of the image of the point in the left space, xR the abscissa of the image of the point in the right space and y the ordinate of the image of the point in the left and right spaces.

4. Device according to claim 3, characterized in that said calculation means (264) are arranged to perform the operations of:

- calculation of the coordinates Xp ^ Yp, and Zp, of the point P and Xp ₂ , Yp ₂ and Zp ₂ of the point P ₂ in a cyclopean space according to the formulas:

X = (D / Δ). (xL + xR)

Y = D. y / Δ

Z = D. f / Δ with Δ = xL - xR - calculation of the differences:

Y _Pl - Yp ₂

- determination, from these three differences, of the distance separating the two points in a three-dimensional space by the formula: d ₁₂ = [(Xp, - Xp ₂ ) ² + (Y _Pl - Yp ₂ ) ² + ( Zp, - Zp ₂ ) ² ] ¹ ' ²

5. Device according to one of claims 1 to 4, characterized in that said communication means comprise a video screen (22) _P ermitting to display an image of the observed field (10).

6. Device according to claim 5, characterized in that said processing means (32) are arranged to generate a significant image of said representative value and superimpose it on the video screen to an image of the field observed.

7. Device according to claim 6, characterized in that said signifying image is an index.

8. Device according to claim 6, characterized in that said signifying image is an area having a color gradient.

5 9. Device according to one of claims 1 to 8, characterized in that it further comprises means (18c) for synchronizing the images of the two cameras.

10. Device according to one of claims 1 to 9, characterized in that it further comprises an analog-digital converter (18d, 18th)

10 interposed between each camera (18a, 18b) and the processing electronics

(20).

1 1. Device according to claim 10, characterized in that it further comprises a memory for storing the signals from the converters.

15 12. Device according to one of claims 10 and 11, characterized in that the processing electronics further comprises selection means (30a) to take into consideration only part of the information from minus one of the converters, in order to reduce the volume of information processed.

13. Device according to one of claims 1 to 12, characterized in that the processing electronics further comprises correction means (30b) for processing said images, in order to reduce the effect of camera aberrations .

14. Device according to one of claims 1 to 13, characterized in that it

25 further includes a switch (23) for limiting the information displayed on the screen to the image from one of the cameras.