WO2001027679A1

WO2001027679A1 - Microscope and method for manufacturing a composite image with a high resolution

Info

Publication number: WO2001027679A1
Application number: PCT/SE2000/001996
Authority: WO
Inventors: Anders Heyden; Johan HÅKANSSON; Björn WALLIN; Adam Karlsson
Original assignee: Cellavision Ab
Priority date: 1999-10-15
Filing date: 2000-10-16
Publication date: 2001-04-19
Also published as: AU1182401A

Abstract

A microscope comprises an object holder (2), a display, optics (4) which in an image plane forms an image of an object (1) which is placed in the object holder (2), and an image sensor (5) for recording the image. The microscope further comprises a display (9), and an input means (10) for selecting an area in an image and a resolution in the area, a storage means and a calculation means which is connected to the image sensor. The calculation means is adapted (i) to digitally store in the storage means at least two recorded images, each of the images partly overlapping at least one other image, (ii) to join the at least two recorded images so that the composite image is an image of the area which is reproduced on the at least two recorded images, and (iii) to store in the storage means (11) information about the composite image. The calculation means is further adapted to show the selected area with the selected resolution on the display (9).

Description

Microscope and method for manufacturing a composite image with a high resolution.

Field of the Invention

The present invention relates to a microscope with an image sensor and a method therefor. Background Art For studying small structures, microscopes are generally used. There are a plurality of different microscop- ing methods, of which the most widespread one is optically imaging microscoping . A common feature of all microscopy methods is that only a limited part of the object can be examined with high resolution on a separate occasion. This area is usually called the field of view of the microscope. The field of view of an optically imaging microscope is determined among other things by the size of the lenses in the optics. The resolution is determin- ed among other things by the numerical aperture of the optics. It is possible to achieve a large field of view and good resolution by using large lenses with a short focal length. However, it is extremely expensive to produce to large lenses, especially if they should also have a short focal length.

As a result, it has only been possible to study relatively small areas on a separate occasion if at the same time high resolution is desirable.

A further problem of conventional microscoping is that it is necessary to study the object through the ocular and turn control means to move the field of view across the object. To sit studying the object through the ocular for a long time is tiring.

One frequently wants to study old preparations on a subsequent occasion. It has then been necessary to file the preparations and put them into the microscope once more, which however is time-consuming work. Moreover, it is not always possible to file the preparations. This applies in particular to preparations in vitro since the preparations must be dried before being stored, which may cause a change of the preparations.

With a view to simplifying handling, it happens that images of relevant areas of a preparation are recorded and stored. It will consequently be possible to study the areas of the images later on without having to file the preparations. This also renders it possible to study relevant areas of wet preparations a long time after the recording of the images . However, when the recorded images are studied, an image of the preparation in its entirety will not be obtained. Nor is it possible to get an idea of how different images of different parts of the preparations are related to each other. Thus there is a need for a microscope which simplifies handling when studying a preparation on a plurality of subsequent occasions. Summary of the Invention

An object of the present invention is to provide a microscope, which minimises the handling of preparations in repeated studies of an individual preparation.

A further object of the present invention is to provide a microscope which is adapted to store images of an area which is larger than the field of view of the micro- scope while at the same time the relationship between different parts of the preparation can be recovered from the stored images .

One more object of the present invention is to provide a method for microscoping, which facilitates repeat- ed studies of an individual preparation.

A further object of the present invention is to provide a memory medium with a computer program, which facilitates repeated studies of an individual preparation.

These objects are achieved by means of a microscope, a memory medium and a method according to the appended claims . A basic concept of the invention is to form a digital image with high resolution of a larger part of the object than has been possible with conventional equipment . A microscope according to the invention comprises an object holder, a display, optics which in an image plane forms an image of an object placed in the object holder, and an image sensor for recording the image. The microscope is characterised in that it further comprises a storage means, an input means for selecting an area in the composite image and a resolution in the selected area, and a calculation means which is connected to the image sensor and which is adapted to digitally store in the storage means at least two recorded images, each of the images partly overlapping at least one further image, to join the at least two recorded images so that the composite image is an image of the area which was reproduced on the at least two recorded images, and to store in the storage means information about the position of the recorded images in the composite image. The calculation means is further adapted to show the selected area with the selected resolution on the display.

By the microscope being adapted to form a digital image of a plurality of partial images, it is possible to later on study an optional area of the image without the preparation needing to be available.

By joining the recorded images is meant that the pixels in the recorded images are given positions in a common system of coordinates . Preferably, the calculation means is adapted to store in the storage means the composite image, but alternatively each of the recorded images can be given a position which is stored in the storage means, the pixels obtaining a common system of coordinates. According to one aspect of the present invention, a computer-readable memory medium is provided, on which a computer program is stored, which is adapted to be used for storing digital images which have been recorded with an image sensor. The computer program comprises instructions for making the computer read an input signal which represents at least two digital images, to join the digi- tal images to obtain a composite image of the area which is reproduced on the at least two digital images, to read information which defines an area and a resolution for the area in the composite image, and to output a digital output signal which represents the selected area with the selected resolution.

It is advantageous to record the partial images with the highest resolution allowed by the optics since the greatest possible freedom of action will then be obtained when later studying the composite image. Preferably the calculation means is adapted to control the object holder. The calculation means moves the object holder between each recording of a new image and stores at least two recorded images in the storage means. The displacement is adjusted so that the recorded images partly overlap each other.

It is advantageous to record the position of the object holder together with the recorded image since such a position can be used when the images are put together. The position of the object holder is obtained from e.g. stepping motors which are used to move the object holder.

According to a less preferred embodiment, the object holder is moved manually. The calculation means is then adapted to receive an input signal with information about the relationship between the recorded images so that it can join the images in the correct manner. Manual precision displacement of the object holder, however, is very time-consuming, and therefore automatic displacement is preferred.

The joining of the images can be carried out in various ways. According to a preferred embodiment of the invention, the calculation means is adapted to join two recorded images by moving the images in relation to each other so that an edge area of one image is fitted to an edge area of the other image, said edge areas overlapping each other. Consequently, high precision in the motion of the object holder is not necessary, which makes it possible to use a simpler and, thus, less expensive object holder.

The recorded images are stored either as black-and- white images or colour images. In both cases, each pixel is associated with at least one value which describes colour or luminous intensity. It has been found advantageous to fit the edge areas to each other by minimising the sum of squares of the difference between a value for each of the pixels in one edge area and the value of the overlapping pixel in the other edge area. The calculation is carried out for at least two different relative positions between the images, whereupon the position which yields the smallest sum of squares is selected as the correct position.

To minimise the calculation work when the images are to be fitted to each other, it has been found advantageous to carry out the fitting of two images in at least two steps, the resolution for the edge areas increasing for each step. As a starting point for the fitting, use is made of approximate positions which have been recorded with the images. The resolution of the recorded image is reduced digitally, whereupon the images are fitted together. The square of the differences between the pixels is summed up with the images in at least two different relative positions. Subsequently, a higher resolution is used, whereupon the fitting-together is repeated for this higher resolution. The optimal position from the first fitting-together is used as a starting point for the second fitting-together .

To further minimise the calculation work, it has been found advantageous to spatially filter the images before putting them together. This means that the effect of, for example, grains of dust which only appear on one of the images, is minimised. The filtering improves the possibilities of minimising the sum of squares by minimising the sum of squares in one direction at a time. Minimising the sum of squares in one direction at a time reduces the calculation work compared with the case where the sum of squares is calculated for all possible relative positions in a predetermined number of pixels from the starting position. However, with unfiltered images it is not always possible to obtain the minimum sum of squares by minimising the sum of squares in one direction at a time.

When the images are stored as colour images, the difference between two pixels can be defined as the sum of the differences for each of the colour compo- nents included. Alternatively, the difference between two pixels can be defined as the difference between the intensity values of one of the colours included.

The put -together image has areas that are present in more than one of the recorded images . The value of the pixels in the put-together image is advantageously an interpolation between the value of corresponding pixels in the recorded images, the value of each of the pixels being weighted in relation to how close to the edge of the respective images they are located. Thus, pixels located far away from the edge are suitably taken into greater consideration than corresponding pixels located close to the edge of another recorded image.

According to one aspect of the present invention, the calculation means is adapted to select edge areas having high spatial frequencies and a high contrast and to join the recorded images by fitting-together the selected edge areas. By selecting edge areas with high spatial frequencies and a good contrast, a good fitting- together is obtained while at the same time the calcula- tion work is minimised.

When more than two images are to be put together, there are a plurality of possible orders in which the images are put together. It is then advantageous for the microscope to be adapted to recognise high spatial frequencies with a high contrast in the edge areas and to put together pairs of the recorded images in an order determined by the spatial frequencies and the contrast at the edge of the images. Thus, the composite image is increased by a recorded image in each step. Consequently not all edges between the different images are adjusted in the composite image. This gives the advantage that the result in the composite image will be less sensitive to structureless areas in the edge area of an image.

Of course, it is possible within the scope of the invention to adjust all edges between all recorded images at the same time. However, extensive calculations are necessary to achieve simultaneous adjustment of all recorded images .

If all images are to be adjusted to each other at the same time it is, however, advantageous to calculate a comparative value for a plurality of different relative positions for each pair of adjoining images, the comparative value being a measure of how well the edge areas correspond to each other, and subsequently put together the recorded images in relative positions, so that a weighted sum of the comparative values for the pairs of images is maximised.

By this method, the calculation work is minimised while at the same time a better total fitting-together of the images is obtained. By weighted sum is meant that the different comparative values are weighted according to how reliable they are.

It has been found advantageous that the comparative values are based on the covariance of the areas where the images overlap each other for a number of different relative positions between the images. In order to further reduce the calculation work, it is advantageous for the comparative values to be calculated for displacements of the images in two orthogonal directions, the images being projected onto rows and columns respectively in the images before the comparative values are calculated.

By calculating comparative values for one direction at a time, the calculation work is significantly reduced. Projection of the images implies that the intensity values in the rows and columns respectively are added up.

It has also been found advantageous to calculate the sum of the comparative values in each of two orthogonal directions at a time. This significantly reduces the calculation work.

Each of the pixels has at least one intensity value. Preferably the calculation of comparative values is carried out by calculating the covariance of the overlapping areas.

When putting together images and calculating comparative values, it is advantageous to carry out the calculations at a gradually increasing resolution. In a first step, images having a lower resolution are then generated digitally, starting from the recorded images. This reduces the calculation work.

After putting together the images, a total image is obtained which reflects the entire recorded area.

It goes without saying that the above features can be combined in the same embodiment.

With a view to further elucidating the invention, detailed embodiments of the invention will be described below without, however, the invention being considered to be restricted thereto. Brief Description of the Drawings

Fig. 1 illustrates a microscope according to a preferred embodiment of the present invention.

Fig. 2 is a flow chart of the method according to a preferred embodiment of the present invention. Fig. 3 is a schematic view of images recorded by means of a microscope according to an embodiment of the present invention. Fig. 4 shows two images recorded by means of a microscope according to an embodiment of the present invention.

Detailed Description of the Invention Fig. 1 illustrates a microscope according to a preferred embodiment of the present invention. An object 1 is placed on a movable object holder 2 and illuminated from below by a bulb 3. An objective 4 images the object on a CCD 5 which has a number of image elements 6 at a mutual centre distance D. The CCD 5 is intended for recording of images and is connected to a calculation means 7 which in this case is a computer 8 with a display 9 and an input means 10 in the form of a keyboard and a mouse. Moreover, the computer is connected to and can control the object holder 2.

A user of the microscope initiates recording of an image of the object 1. In the initiation, a user inputs via the input means 10 information about how the image is to be recorded, i.e. the desired size of it, the desired resolution and how the object holder is to be arranged in relation to the objective at the beginning of the recording. The object 1 is, for example, a specimen of white blood cells which is to be studied in the microscope. By means of the inputted information, the microscope records images that are stored in a storage means 11 in the computer 8. Between each recording and storing of an image, the computer 8 controls the object holder 2 to be moved so as to image a new area, which partly overlaps the preceding area that has been recorded. The storage means 11 can be a direct access memory or a hard disk in the computer 8. Thus, a number of images are recorded and stored, which correspond to the area that was defined in the inputting via the input means 10.

Fig. 2 is a flow chart of the function of a micro- scope according to a preferred embodiment of the present invention. In step 12, information about which area is to be studied in the microscope is inputted via the input means. With the aid of the inputted information, the microscope records in step 13 a number of images which at least partly overlap each other. In step 14, the recorded images are put together so that a composite image forms, which covers the entire area that was covered with the recorded images. In step 15, the composite image is shown on the display 9 of the calculation means. In step 16, a user inputs information about which area in the shown image is to be shown at higher magnification, the magnified image also being inputted in step 16. In step 17, the selected area at the selected magnification is shown on the display 9. Of course, the image in step 17 cannot be shown with a better resolution than that recorded in step 13. However, in step 17 it is possible to show the image at an optional magnification. According to the preferred embodiment, the images are recorded in step 13 with maximum resolution to allow as great freedom of action as possible in steps 16 and 17. The image shown in step 15 does not always reproduce the maximum resolu- tion that is found in the images recorded in step 13 since the size of the display is limited.

Fig. 3 shows a number of images such as 18 and 19 that have been recorded in step 13. The edge areas are graded in respect of high spatial frequencies and a high contrast. The images recorded in step 13 are put together to a composite image 20. When the recorded images are put together, they are put together in pairs in an order which is determined by how they have been graded with respect to contrast and the presence of high special fre- quencies. For example, one begins with image 18, to which image 19 is added. When images 18 and 19 are to be joined, an edge area of image 18 is compared with an image area of image 19 so that they correspond with each other within the area 21 in which they overlap each other. Sub- sequently, the remaining images are added in an order which is determined by which edge has the best contrast and the greatest presence of high spatial frequencies. When all registered images have been put together, the composite image is shown on the display 9. A user inputs via the input means information about which area in the composite image 20 is to be shown at higher magnifica- tion. The selected area can be, for example, the area 22, which contains information from four recorded images. In the areas of the composite image 20 which contain information from more than one image, there is made in each pixel an interpolation between the values in the pixels from each of the recorded images, weighting being carried out so that pixels located close to the edge of a recorded image obtain less weight than such pixels as are located further away from the edge of another image.

Fig. 4 shows two images 23 and 24 which have been recorded in step 13. With reference to Fig. 4, the putting-together of two recorded images will now be described in more detail. Images 23 and 24 have been recorded with a certain overlap and their fitting-together is carried out by making a comparison within an area 25 so that correspondence is obtained between the two images in the area 25. By means of the computer 8, the area 25 is selected, in which there is a great presence of high special frequencies and a high contrast. In the area 25 there are structures that are to be found on both images, such as the structures 26 on the image 23 and the structures 27 on the image 24. The resolution in the images 23, 24 is reduced electronically by a factor 8, so that the number of pixels in the images is reduced by a factor 64. The image 24 is then moved in relation to the image 23 so that the sum of squares of the differences between the value in one pixel in the image 23 in the area 25 and the overlapping pixel in the image 24 in the area 25 is minimised. The sum of squares of the differences is in most cases minimal when the image 24 has been moved so that the structures 26 have the same position as the structures 27. Then the minimising is repeated for a gradually increasing resolution when the obtained relative position is used as a starting position for the next minimising step. The resolution is increased by a factor 2 in each minimising step until the maximum resolution has been achieved. As a result, the images are placed gradually closer to their correct relative position. In each minimising step, the images are moved to different positions, which represent a combination of the orthogonal displacements from the starting position. The image 23 is composed of a number of pixels which are each asso- ciated with a value describing the luminous intensity of the pixel. If the image 23 is a colour image, it is associated with three different colours, red, green and blue, which each have a value describing the intensity of precisely that colour in the pixel. If the images are colour images, the comparison is made in the edge area between each of the colours or between a converted image, in which case the converted image is a grey-scale image of the original colour image. It is also possible to use only one of the colours when comparing the images . With reference to Fig. 3, an alternative method of putting together the images will now be described. First the optimal fitting between each pair of images 18, 19 is determined. The fitting-together of two images is carried out by calculating the covariance of the area within which the images overlap each other for a plurality of different relative positions between the images 18, 19. The covariance is used as a comparative value to find the best fitting-together of the images. Before calculating the covariance, a projection of the images is made onto each of two orthogonal directions in the images. Thus, the rows and the columns respectively are added up, after which the covariance is calculated in the direction of the columns and rows respectively for a number of different relative positions. The covariance is calculated in the usual fashion except that no dividing by the number of measuring points occurs. For each of the pairs of images a set of covariances is obtained as a function of the displacement. In each of the sets containing covariances for different displacements, its maxima are searched out. This point and two adjoining points are used to form a curve of the second degree, which together with the corresponding curves of the second degree for the other sets is used to put together all the recorded images. When fitting together the pairs of images, a weight W is calculated which is a measure of how reliable the fitting-together is. The weight W is given by:

W S0)- 0.50*max(S)^'

v 0.50*max(S)

wherein

i e [s(i) ≥ 0.25max(S),|/- max| > 2]- j e [s(j) ≥ 0.50max(S),|_./ -max| = l],

wherein S(i) is the covariance at the displacement i, and S (max) = max(S) . If W becomes negative, the fitting- together is considered unusable and W is set at zero. If the images are not square, it is easier to find a good fitting-together of between the images along the long edge. Then the fitting-together is advantageously carried out in this direction first. After determining the relative positions of each pair of images, all the recorded images are put together so that a weighted sum of the comparative values is maximised. This means that a weighted sum of the covariances for the different pairs of images is maximised. This, too, is made first in one of the direction of the columns and the direction of the rows and then in the other direction. The maximisation is carried out by minimising the function

E = -fo* - *_MJi ,

wherein m is the number of images in a column, n the number of images in a row, Cι,_k/0 is the optimal fitting- together of image [l,k] and its right neighbour and Cι,_k,ι is the optimal fitting-together of image [l,k] and its lower neighbour. Wι_;k,₀ and Wι_;krι are correspondingly the weight of the respective two-image fittings. Xχ_;k is the displacement of the image in the x direction or in the y direction depending on the optimisation that has been carried out. A person skilled in the art realises that the invention is limited to the embodiments described, which can be varied within the scope of the invention. For example, the CCD can be replaced by a vidicon. Moreover the user need not necessarily initiate the recording of the par- tial images by specifying the size and resolution of the completed image since this can be carried out according to a predetermined schedule stored in the computer.

Claims

1. A microscope comprising an object holder (2), optics (4) which in an image plane forms an image of an object (1) placed in the object holder (2), and an image sensor (5) for recording the image, c h a r a c t e r i s e d in that the microscope further comprises a display (9) , an input means (10) for selecting an area in an image and a resolution, a storage means (11) , and a calculation means (8) which is connected to the image sensor (5) and which is adapted

(i) to digitally store in the storage means at least two recorded images, each of the recorded images compris - ing a plurality of pixels, and each of the images partly overlapping at least one other image,

(ii) to join the at least two recorded images so that the composite image is an image of the area which is reproduced on the at least two recorded images (18, 19, 23, 24) , and

(iii) to store in the storage means (11) information about the composite image, and to show, on the display (9) , the area having the selected resolution and selected with the aid of the input means.

2. A microscope as claimed in claim 1, c h a r a c t e r i s e d in that the calculation means is adapted to store the composite image in the storage means

(11) .

3. A microscope as claimed in claim 1, c h a r a c t e r i s e d in that the calculation means is adapted to store in the storage means (11) information about the positions of the recorded images in the composite image .

4. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the cal- culation means further is adapted to control the motion of the object holder (2) and to store at least two recorded images in the storage means (11), between which recordings the object holder (2) has been moved so that the recorded images partly overlap each other.

5. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the calculation means (7) is adapted to receive an input signal with information about the relationship between the recorded images.

6. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the calculation means (7) is adapted to join two recorded images by placing the images in relation to each other so that an edge area of one image is fitted to an edge area of the other image, said edge areas overlapping each other.

7. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the calculation means (7) is also adapted to calculate a compa- rative value for a plurality of different relative positions for each pair of recorded images when they are arranged so that an edge area of one image overlaps an edge area of the other image, the comparative value being a measure of how well the edge areas correspond to each other, and subsequently put together the recorded images in relative positions so that a weighted sum of the comparative values for the pairs of images is maximised.

8. A microscope as claimed in claim 7, c h a r a c t e r i s e d in that comparative values are calcu- lated for displacements of the images in two orthogonal directions, the images being projected onto rows and columns respectively in the images before the comparative values are calculated.

9. A microscope as claimed in claim 7 or 8 , c h a r a c t e r i s e d in that the calculation of the sum of the comparative values is carried out in each of two orthogonal directions at a time.

10. A microscope as claimed in claim 7, 8 or 9, c h a r a c t e r i s e d in that each of the pixels has at least one intensity value, and that the calculation of the comparative values is carried out by calculation of the covariance of the overlapping edge areas .

11. A microscope as claimed in claim 6, c h a r a c t e r i s e d in that each of the pixels has at least one intensity value, and that fitting-together of the edge areas results in minimising of the sum of squares of the difference between the intensity value for each of the pixels in one edge area (25) and the overlapping pixel in the other edge area.

12. A microscope as claimed in claim 6 or 11, c h a r a c t e r i s e d in that the calculation means is adapted to select, for said fitting-together, edge areas in the images with a great presence of high spatial frequencies and a high contrast.

13. A microscope as claimed in any one of claims 6, 11 or 12, c h a r a c t e r i s e d in that the calculation means is adapted to recognise structures in the edge areas (25) and to join the recorded images in an order determined by the spatial frequencies and the contrast at the edge of the recorded images.

14. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the recorded images are recorded at maximum resolution of the optics (4) .

15. A method for microscoping comprising the step of recording with an image sensor (15) at least two images, which partly overlap each other and which have been reproduced in a microscope, c h a r a c t e r i s e d in that it further comprises the steps of joining the recorded images (18, 19, 23, 24) to obtain a composite image of the area which is reproduced on the at least two recorded images (18, 19, 23, 24), selecting an area and a resolution for the area in the composite image, and showing the selected area with the selected resolution on a display (9) .

16. A method as claimed in claim 15, c h a r a c t e r i s e d in that it further comprises the steps of selecting on the images pairs of the edge areas which overlap each other, and placing the edge areas in relation to each other so that the sum of squares of the differences between each of the pixels in the edge area of one image and the overlapping pixel in the edge area of the other image is minimised.

17. A method as claimed in claim 15, c h a r a c t e r i s e d in that it further comprises the steps of selecting in pairs on the images the edge areas which overlap each other, calculating a comparative value for a plurality of different relative positions for each pair of images when the edge areas overlap each other, the comparative value being a measure of how well the edge areas correspond to each other, and subsequently joining the recorded images in relative positions so that the sum of the comparative values for the pairs of images is minimised.

18. A method as claimed in claim 17, characterised in that the comparative values are calculated for dis- placements of the images in two orthogonal directions, the images being projected onto the rows and columns respectively before the comparative values are calculated.

19. A method as claimed in claim 17 or 18, c h a r a c t e r i s e d in that the minimising of the sum of the comparative values is made in each of two orthogonal directions at a time.

20. A method as claimed in claim 16, c h a r a c t e r i s e d in that the placing of the images, and the associated calculation of the sum of the differences between their pixels, are carried out in at least two steps, the resolution in the image increasing in each step .

21. A method as claimed in any one of claims 16-20, c h a r a c t e r i s e d in that the recorded images are spatially filtered before being put together.

22. A computer-readable memory medium, on which a computer program is stored, which is intended to be used for storing of digital images which have been recorded with an image sensor, c h a r a c t e r i s e d in that the computer program comprises instructions for making the computer (8) read an input signal which represents at least two digital images, join the digital images (18, 19, 23, 24) to obtain a composite image of the area which is reproduced on the at least two digital images (18, 19, 23, 24), read information which defines an area and a resolution for the area in the composite image, and output a digital output signal which represents the selected area with the selected resolution.

23. A computer-readable memory medium as claimed in claim 22, c h a r a c t e r i s e d in that the computer program also comprises instructions for making the computer (8) select on the images pairs of edge areas (25) which overlap each other, and place the images in relation to each other so that the sum of the differences between each of the pixels in the edge area of one image and the overlapping pixel in the edge area of the other image is minimised.

24. A computer-readable memory medium as claimed in claim 22, c h a r a c t e r i s e d in that the computer program also comprises instructions for making the computer (8) perform the placing of the images and the associated calculation of the sum of the differences between their pixels in at least two steps, the resolution in the image increasing in each step.