WO2008152605A1 - Multi-spot scanning optical device for imaging of a sample - Google Patents

Multi-spot scanning optical device for imaging of a sample Download PDF

Info

Publication number
WO2008152605A1
WO2008152605A1 PCT/IB2008/052338 IB2008052338W WO2008152605A1 WO 2008152605 A1 WO2008152605 A1 WO 2008152605A1 IB 2008052338 W IB2008052338 W IB 2008052338W WO 2008152605 A1 WO2008152605 A1 WO 2008152605A1
Authority
WO
WIPO (PCT)
Prior art keywords
array
light spots
sample
detector
image
Prior art date
Application number
PCT/IB2008/052338
Other languages
French (fr)
Inventor
Dirk L. J. Vossen
Levinus P. Bakker
Original Assignee
Koninklijke Philips Electronics N.V.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics N.V. filed Critical Koninklijke Philips Electronics N.V.
Publication of WO2008152605A1 publication Critical patent/WO2008152605A1/en

Links

Classifications

    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00Microscopes
    • G02B21/0004Microscopes specially adapted for specific applications
    • G02B21/002Scanning microscopes
    • G02B21/0024Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/0032Optical details of illumination, e.g. light-sources, pinholes, beam splitters, slits, fibers
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00Microscopes
    • G02B21/0004Microscopes specially adapted for specific applications
    • G02B21/002Scanning microscopes
    • G02B21/0024Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/008Details of detection or image processing, including general computer control
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00Microscopes
    • G02B21/36Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365Control or image processing arrangements for digital or video microscopes
    • G02B21/367Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison

Definitions

  • Multi-spot scanning optical device for imaging of a sample
  • the invention relates to multi-spot scanning imaging, in particular to a multi- spot scanning microscope.
  • a scanning microscope is used to image a sample by scanning a focused spot of light (light spot) across the sample and collecting, on a detector, the light interacted with the sample, the interaction being, for instance, reflection, transmission or scattering.
  • the light spot is obtained by focusing a light beam on the sample using a microscope objective.
  • Multi-spot scanning microscopes have been developed.
  • a multi-spot scanning microscope a plurality of light beams are simultaneously irradiated on the sample, forming an array of focused light spots. It is the whole array of light spots that is scanned within the sample, the different focused light spots being spaced apart the ones from the others by a fixed distance and displaced all together.
  • a multi-spot scanning microscope permits to scan (displace) the array of light spots on distances inferior to the dimensions of the sample, since the light spots of the array are distributed on a certain surface of the sample or even on the whole surface of the sample, in which latter case the array only needs to be displaced on the distances corresponding to the surface defined by four proximal light spots in order to cover the whole surface of the sample, all the light spots being displaced together.
  • an image of the spots irradiated in the sample is recorded on a detector, which therefore records an array of image light spots.
  • a complete image of the sample is constructed by processing the various recorded images, the processing being performed once all the images have been recorded or in real time; in this latter case the complete image is completed as the images are recorded. Therefore, when the array of light spots is scanned with respect to the detector, the corresponding positions of the focused light spots in the sample have to be determined, in order to build a correct complete image from the images recorded at the different positions.
  • image should be understood in a broad sense.
  • An image comprises information on the different irradiated points of the sample. The information may contain the amount of reflected light at each point, the amount of scattered light at each point, etc. Imaging permits to measure the spatially extended structural properties of the sample.
  • image will be used to designate:
  • the microscope comprises means for displacing the array of light spots with respect to the sample.
  • Each recorded image corresponds to a particular position of those displacing means and therefore to a corresponding position of the array of light spots within the sample.
  • the displacing means are displaced step by step by predetermined distances and an image is recorded at each predetermined position, or the displacing means are displaced continuously (for instance at a constant speed) and an imaging activator records images at each predetermined position.
  • Each image is attributed to a predetermined position in the sample.
  • the displacing means and/or the imaging activator may suffer from lack of accuracy. If the array of light spots is not at the correct predetermined position when the image is taken, the information on the detector will be attributed to incorrect points in the sample, therefore leading to a distorted image.
  • measuring means may be connected to the means for displacing the array of light spots, so as to measure their actual displacement; however, such a solution is not necessarily very accurate and is not easy to implement.
  • Another solution is to have very accuracy displacing means that can be trusted in; however, such displacing means are expensive.
  • Another solution for determining the position of the array of light spots in the sample consists in performing a measurement of the position of an element of the microscope, which position is related to the relative movement between the array of light spots and the sample.
  • the chosen element can be the holder of the sample, if this latter is moved during the scanning operation (the array of light spots being fixed); in such a case, the position of a particular array of light spots in the sample is deduced from the position of the holder at the time the particular array was irradiated on the sample. Again, this solution is not very accurate.
  • a method for imaging a sample with a multi-spot scanning microscope comprising:
  • the invention proposes to attribute, to relative positions of the light spots on the detector between the images, relative positions of the light spots in the sample.
  • the method of the invention it is not the absolute position of the array of light spots that is looked for (even if this is possible according to an embodiment of the invention) but the relative position, that is to say, the position relatively to the other positions, in other words, the evolution of the positions between the images.
  • the evolution of the positions depends on the actual positions of the means for scanning the array of light spots in the sample when the images are recorded. Therefore, by calculating the relative positions of the array of light spots in the sample, when reconstructing the complete image of the sample on the basis of the images recorded on the detector, the recorded images are attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample.
  • the positions - to which the recorded images are attributed - are placed the ones relatively to the others, i.e. the positions are spaced apart the ones from the others, by taking into account their calculated relative positions.
  • This permits to avoid distortions of the complete image; indeed, distortions would occur if the recorded images were attributed to wrong positions of the array of light spots in the sample relatively the ones to the others.
  • the result of the method of the invention is that the proportions of the sample are kept in the complete image that is constructed.
  • the calculated relative positions of the array of light spots in the sample are the same as the corresponding calculated relative positions of the array of image light spots on the detector.
  • the relative positions of the array of image light spots on the detector are calculated on the basis of the relative positions of a plurality of light spots of the array that are averaged over the image light spots of the array.
  • the method comprises:
  • the method comprises:
  • the method comprises:
  • the method comprises:
  • the optical device comprising a pixelated detector, which comprises a plurality of pixels adapted to be activated by the image light spots, the position of an image light spot is calculated as the position of at least one pixel activated by said image light spot.
  • an image light spot activating a plurality of pixels the position of the image light spot is calculated as the average position of the activated pixels.
  • the pixels being activated at a certain intensity depending on the intensity of the corresponding image light spot the average position of the activated pixels is weighted by the respective intensity of their activation.
  • the distances on the detector are calibrated, that is to say that, to a distance on the detector, calculated in a number of pixels, an actual distance on the sample is attributed.
  • the relative positions of the array of image light spots are calculated as the average positions of the image light spots on the detector, in arbitrary units.
  • the array of light spots comprising spots aligned in two directions, the positions into which the array of light spots is scanned substantially form a straight line that makes an angle with the directions of the array.
  • the array of light spots comprising spots aligned in two directions, the positions into which the array of light spots is scanned substantially form a set of straight segments, each straight segment having a length substantially equal to the distance between two neighboring light spots in the array and making substantially no angle with one direction of the array, the adjacent straight segments being joined by intermediate segments.
  • a multi-spot scanning microscope comprising:
  • the device comprises a first pixelated detector with high magnification and a second pixelated detector with low magnification.
  • the device comprises means for blurring the image of the image light spots on the detector.
  • the device is a multi-spot scanning microscope.
  • a method for processing a set of recorded images in order to form an image of a sample each recorded image representing an array of light spots on a detector forming the image of an array of light spots irradiated in the sample, each recorded image corresponding to a different position of the array of lights spots in the sample, the method comprising:
  • the images on the detector may be recorded on a suitable medium and the set of recorded images may be provided to data processing means carrying out the above method.
  • - Fig.1 is a schematic diagram of a multi-spot scanning microscope according to the invention.
  • - Fig.2 shows an image, on the detector of the microscope of Fig.1, of an array of light spots irradiated in the sample;
  • - Fig.3 shows a zoomed- in image of a single image light spot at a first position on the detector
  • - Fig.4 shows a zoomed-in image of the single image light spot of Fig.3 at a second position on the detector.
  • an embodiment of an optical scanning microscope 1 comprises a light source (not represented), generating a parallel light beam 2 propagating along the optical axis z of the microscope 1.
  • the light beam 2 is incident on a diffractive structure 3, such as a binary phase structure.
  • the diffractive structure 3 generates a multitude of orders of diffraction in numerous directions, the orders interfering and creating at a certain distance an array 4 of light spots S.
  • the light source and the diffractive structure 3 form means for generating an array of light spots 4. Any other suitable means may be contemplated for generating an array of light spots 4.
  • the focused light spots S of the array 4 are irradiated on a sample 5, which is positioned on a sample support 6, herein comprising for example two glass plates (or slides), as known in the art.
  • the irradiated light spots S form an array 4 in two perpendicular directions, for instance in x and y directions or two perpendicular directions transverse to the direction z of the optical axis and having an angle with the x and y directions.
  • Each light spot is separated from the nearest light spots by a pitch, which is in this embodiment equal in the two directions of the array 4.
  • other orderings of the light spots within the array 4 may be contemplated, notably more complex configurations.
  • the irradiated light spots interact with the sample 5 and are imaged on a pixelated detector (sensor) 8 onto which they are focused by imaging optics 9, thereby forming on the detector 8 an array 7 of image light spots S'.
  • the structure of the imaging optics will not be detailed as it is easily accessible to the person skilled in the art.
  • An image light spot S' should be understood as the image, on the detector 8, of a light spot S irradiated in the sample 5.
  • the detector 8 is provided on the same side as the side of incidence of the array of irradiated light spots 4, with appropriate optics for irradiation and detection of the light spots on the same side of the sample 5.
  • the detector 8 is provided on the opposite side, the light passing though the sample 5 and its support 6.
  • each light spot S irradiated on the sample 5 is imaged into a single image light spot S' on the detector 8. Therefore, the array 4 of irradiated light spots S is imaged into an array 7 of image light spots S', the two arrays 4, 7 having a global similar shape, an example of which can be seen on Fig.2 which shows image light spots S' on the detector 8.
  • Each image light spot S' is analyzed individually to measure a particular feature, for instance its intensity, in order to deduce therefrom a particular characteristic of the point of the sample 5 corresponding to the image light spot S', that is to say, the point of the sample 5 that has been irradiated by the light spot S which, interacted with the sample 5, has been imaged as the image light spot S' on the detector 8.
  • a particular characteristic of the point of the sample 5 corresponding to the image light spot S' that is to say, the point of the sample 5 that has been irradiated by the light spot S which, interacted with the sample 5, has been imaged as the image light spot S' on the detector 8.
  • attenuation in the intensity of an image light spot S' may reveal the presence of an absorbing substance at the corresponding point of the sample 5.
  • the array of irradiated light spots 4 may be scanned into a plurality of positions in the sample 5. At each position of the array 4 in the sample 5, a corresponding image is recorded on the detector 8; the recorded image comprises the array 7 of image light spots S'. An example of such a recorded image is shown on Fig.2. Each image light spot S' of this array 7 comprises information on the sample 5, as explained above.
  • a complete image of the sample 5 (or of a certain surface of the sample 5) is constructed on the basis of all the recorded images. Indeed, if the array of irradiated light spots 4 has been scanned into a plurality of positions within all the sample 5 (or within all said certain surface of the sample 5), an image being recorded at each position, the plurality of images contain image light spots S' representing all the positions in the sample 5 (or all the positions in said certain surface of the sample 5). Constructing the complete image may be performed after all the images have been recorded or in real time as the images are recorded.
  • the microscope 1 comprises means for scanning the array 4 of light spots S irradiated in the sample 5.
  • those means may comprise means for displacing, relatively to the sample 5, the means for generating the array of irradiated light spots 4.
  • the microscope 1 comprises means for displacing the diffractive structure 3 relatively to the sample 5.
  • the array of irradiated light spots 4 is scanned along two directions, for instance the two directions of the array, into a plurality of positions, as explained above; the displacing means may be displaced whether step by step or continuously.
  • the array 4 only needs to be scanned over the surface of an elementary cell of the array 4, defined by four proximal light spots S, to scan the whole sample surface; indeed, since all the irradiated light spots S are displaced together, if each spot covers the elementary cell, then the array 4 covers the whole scanned surface of the sample 5.
  • the array 4 may be scanned over the surface of the sample 5 that is in the field of view of the imaging optics 9 of the microscope 1 and an image is taken from that surface, and optionally the sample 5 is displaced so as to scan another surface of it.
  • the array of irradiated light spots 4 is scanned relatively to the sample 5 in such a manner that each required point of the sample 5 (depending on the required resolution) is touched at least once by a light spot S, in one of the scanning positions.
  • the resolution is determined by the size of the light spots and the distance between two positions of the light spots, that is to say, the distance between two successive positions.
  • the light spots can be of a diameter equal to l ⁇ m, spaced apart in the array by 15 ⁇ m, the array being moved by l ⁇ m between two positions of the array 4 so as to scan the entire surface of the sample 5.
  • the array can also be displaced in such a way that the entire surface of the sample 5 is not scanned but only a discrete portion of it is scanned, for instance if 1 ⁇ m diameter light spots are displaced, between two successive positions, by 2 ⁇ m.
  • oversampling is performed, that is to say, the displacement steps between two successive positions of the light spots are smaller than the diameter of the light spots, for instance l ⁇ m diameter light spots are displaced, between two successive positions, by 0.5 ⁇ m; oversampling permits to get a better resolution.
  • a one direction translation scanning may be sufficient to scan all the positions in the sample 5, for instance if two adjacent light spots are offset, preferably of half the diameter of a light spot (such a configuration leads to an image, on the detector 8, comparable to the one of Fig.2, where it should be contemplated that the array is displaced horizontally between two positions).
  • the array (4) of light spots (S) comprising spots (S) aligned in two directions, the positions into which the array (4) of light spots (S) is scanned substantially form a straight line that makes an angle with the directions of the array
  • the array (4) of light spots (S) comprising spots (S) aligned in two directions, the positions into which the array (4) of light spots (S) is scanned substantially form a set of straight segments, each straight segment having a length substantially equal to the distance between two neighboring light spots (S) in the array (4) and making substantially no angle with one direction of the array (4), the adjacent straight segments being joined by intermediate segments; those intermediate segments may be parallel to the second direction of the array.
  • Fig.l one particular irradiated light spot S is represented, in three successive positions Pl, P2, P3 of the array of light spots 4 in the sample 5 (Fig.l is not on scale, for the sake of clarity). From the first position Pl to the second position P2 of the light spot S in the sample 5, the array 4 of irradiated light spots S has been displaced by a first displacement distance dl. From the second position P2 to the third position P3 of the light spot S in the sample 5, the array 4 has been displaced by a second displacement distance d2 in the sample 5.
  • the light spot S irradiated on the sample 5 is imaged on the detector 8 as an image light spot S'.
  • Each position Pl, P2, P3 of the irradiated light spot S in the sample 5 results in a position P'l, P'2, P'3 of the image light spot S' on the detector 8.
  • the array 7 of image light spots S' has been displaced by a first displacement distance d'l.
  • the array 7 has been displaced by a second displacement distance d'2 on the detector 8.
  • the three successive images on the detector 8, corresponding to said positions Pl, P2, P3, should therefore not be attributed to points in the sample 5 spaced apart by equal distances, otherwise distortions in the constructed complete image will occur.
  • a relative position is a position compared to something, that is to say, relatively to something, here, relatively to the other positions.
  • the relative positions are calculated on the basis of the distances between two successive positions.
  • the optical device is a multi-spot scanning microscope comprising a pixelated detector 8, which comprises a plurality of pixels that may be activated by the image light spots S'
  • the relative positions of the image light spots S' are calculated as the relative positions of the pixels activated by the corresponding image light spots S'. Indeed, an image light spot S' activates at least one pixel, with a certain intensity.
  • the displacement distances d'l, d'2 between successive positions of the image light spots S' on the detector 8 are used to calculate the relative positions of the array 4 of light spots S in the sample 5.
  • a relative position is calculated as the ratio between successive distances d'l, d'2 between successive positions P'l, P'2, P'3.
  • the relative positions of the array 7 of image light spots S' between successive images are here attributed to the array 4 of irradiated light spots S, that is to say, the relative positions of the array 4 of irradiated light spots S in the sample 5 are set equal to the ratios calculated for the array 7 of image light spots S'.
  • Fig.3 and 4 each represent a position P'l, P'2 of a single image light spot S' on the detector 8. Each position P'l, P'2 corresponds to an image recorded on the detector 8. Those positions P'l, P'2 are successive positions, corresponding to successive images. Let us suppose that, between the first and second positions P'l, P'2, there is a displacement distance d'l of 12 pixels. Let us also suppose that between the second and third position P'2, P'3 (not shown on Fig.3 and 4 but shown on Fig.l), there is a displacement distance d'2 equal to 15 pixels. From this distance calculation, it can be concluded that the second displacement distance d'2 is 25% bigger than the first displacement distance d'l.
  • the ratio between the distances d'l, d'2 on the detector 8 is therefore equal to 1.25.
  • This ratio is applied to the corresponding positions Pl, P2, P3 (of the array 4 of irradiated light spots S) that are attributed to the recorded images when constructing the complete image of the sample 5.
  • the images respectively corresponding to the first, second and third positions P'l, P'2, P'3 on the detector 8 are attributed to positions Pl, P2, P3 in the sample 5 observing the calculated relative positions, that is to say, the calculated ratio of the distances between the positions P'l, P'2, P'3. Therefore, the distance d2 between the second and third positions P2, P3 is 1.25 times the distance dl between the first and second positions Pl, P2. Distortions are hence compensated for.
  • Imaging can therefore be performed, which measures the spatially extended structural properties of the sample 5.
  • the relative positions are calculated on the basis of the distances between successive positions, between successive images. Other relative positioning can of course be contemplated.
  • the relative positions of the array 7 of image light spots S' are calculated as the averaged positions of the light spots S', in arbitrary units.
  • the positions of all the image light spots S' are calculated and those positions are averaged in order to give a certain position, which is the average position of the spots. For instance, if each image light spot S' on the detector 8 has coordinates (x,y), the x values of all the image light spots S' are summed and averaged and the y values of all the image light spots S' are summed and averaged, those averaged values giving an average position in arbitrary units.
  • the relative positions of the array 4 of irradiated light spots S in the sample 5 can be determined.
  • the arbitrary units can in fact be considered as relative coordinates. It should be noticed that, in the calculation of the averaged position in arbitrary units, the image light spots S' situated in the outer bands of the array 7 may not be taken into account if there is a risk that, during the scanning, a spot disappears or appears on a side of the detector between two images; at the contrary, the positions can only be compared between the images if the same spots S', having been displaced, are taken as the basis of the determination of the relative position of the array 7.
  • the means for scanning the array 4 of irradiated light spots S may be cheap ones, since even if those means move the array 4 of light spots S by irregular distances, these irregularities will be compensated for thanks to the method of the invention.
  • the method of the invention applies to a relative movement between two assemblies: the means for generating and displacing the array 4 of light spots S on the one hand, the sample 5 and the detector 8 on the other hand. If the sample 5 were moved relatively to the detector 8, the means for generating and displacing the array 4 of light spots S being fixed, the detector 8 would indeed measure information on different positions in the sample 5, but the image light spots S' would appear at the same positions on the detector 8 for every images, therefore forbidding to perform the method of the invention, which establishes a relation between the relative positions on the detector 8 and the relative positions in the sample 5.
  • the relative positions of the array 7 of image light spots S' between the images can easily be calculated on the basis of the relative positions of the corresponding activated pixels.
  • the relative positions of the pixels activated by a unique particular image light spot S' between the images are calculated and attributed to the whole array 7 of image light spots S' and, therefore, to the whole array 4 of irradiated light spots S.
  • the calculations are made for only one image light spot S' between successive images, then considering that the light spots S' of the array 7 of image light spots S' are all displaced by the same distances as this particular light spot S', which are therefore considered as the displacement distance of the array 7.
  • the relative positions of a plurality of image light spots S' (or corresponding activated pixels) between images are calculated, those relative positions being averaged. According to a particular embodiment in that case, the relative positions of all the image light spots S' are averaged.
  • the method comprises:
  • three embodiments can be contemplated. Firstly, the positions of the light spots S' are calculated and averaged, then the distances between successive averaged positions are calculated and their ratio is calculated. Secondly, the positions of the light spots S' are calculated, then the distances between successive positions are calculated and averaged and their ratio is calculated. Thirdly, the positions of the light spots S' are calculated, then the distances between successive positions are calculated and their ratio is calculated and then averaged.
  • the relative positions of the array 7 are considered as being equal to the averaged relative positions of the image light spots S' over the array 7.
  • An advantage of averaging the relative positions of the different image light spots S' of the array 7 is that it provides a better accuracy of the position measurement.
  • the actual positions of the image light spots S' on the detector 8 may be calculated.
  • the total number of pixels N activated per image light spot S' on the detector 8 has to be at least one.
  • a plurality of pixels are activated by each image light spot S', so that the position of the image light spots S' can be calculated with sub-pixel accuracy.
  • the position of an image light spot S' is calculated as the average position of the pixels activated by the image light spot S', weighted by the respective intensities of the activated pixels. The more pixels per spot, the higher the accuracy of the position determination.
  • at least four pixels are activated per image light spot S'.
  • each spot activates a plurality of pixels.
  • the pixels activated by one image light spot S' are determined according to a predetermined criterion, such as a threshold comparison. Then, in order to take into account the weight of these pixels, an intensity is calculated for each of them, and, according to this, an average position is determined.
  • averaging step applies there to the different pixels activated by a single light spot, the different calculated positions possibly being subsequently averaged over the whole array 7 of image light spots S' for calculating the relative positions of the array 7, as explained above.
  • the multi-spot scanning microscope 1 comprises two detectors (not represented), a first pixelated detector with high magnification and a second pixelated detector with low magnification.
  • a subset of image light spots S' are imaged onto the first detector, which ensures enough pixels per spot for accurate position determination.
  • the complete array of image light spots S' is imaged at the second detector, which allows for a large field.
  • the second detector is used for global position determination and the first detector with a subset of light spots with many pixels is used for gathering intensity information used for imaging, as well as for determining precise positions of the image light spots.
  • a beam splitter can be provided that separates the light beams into two paths, one on each detector.
  • the multi-spot scanning microscope 1 comprises means for blurring the image light spots S' on the detector 8.
  • Those means may for instance comprise defocus means or means for creating spherical aberration, as well known in the art.
  • the image light spot size is enlarged on the detector 8, which activates more pixels and permits to determine the position of the spots more accurately, without lowering the resolution of the microscope.
  • the resolution of an image produced with the multi-spot scanning microscope 1 is determined by the quality of the light spots S irradiated in the sample 5, the blurring in the imaging system does not lower the overall resolution of the system.
  • the maximum blur that is allowed is when two adjacent image light spots S' imaged on the detector 8 can still be separated.
  • the blurring does not cause any problem for gathering information from the image light spots, since even if the spot intensity is spread over a larger number of pixels, its value is the same; therefore, by integrating the intensity over all the pixels, the value of the intensity is recovered.
  • the distances on the detector 8 are calibrated, that is to say that an actual distance on the sample 5 is attributed to a distance on the detector 8 (calculated in a number of pixels).
  • the numerical positions and/or distances on the detector 8 can be calculated in order to give a scale to the constructed image of the sample 5.
  • an average can be taken to give the position of the array 4 of light spots S in the sample 5 with very high accuracy (sub lOOnm). Therefore, not only the image is not distorted, but also its scale is known.
  • the method of the invention in particular permits real-time calibration.
  • Another way of attributing a distance on the detector to a distance in the sample is to know the magnification of the imaging optics.
  • Another simple way to calibrate the distances on the detector 8 is to use the known distance between two successive light spots of the array 4 of light spots S irradiated on the sample 5. If this distance is known - which is usually the case since the means for generating the array 4 of light spots S are calibrated - therefore the calibration is easily obtained by calculating the number of pixels separating two successive image light spots S' and attributing, to this number of pixels, the known distance.
  • the images may be recorded on a suitable medium and the set of recorded images may be provided to data processing means, adapted to process the recorded images in order to form a complete image of the sample (5).
  • a computer program may be provided for enabling the data processing means to carry out the processing method.
  • the computer program may be recorded on a computer program product. While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.
  • a computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid- state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Abstract

The invention relates to a method for imaging a sample (5) with a multi-spot scanning microscope, comprising: - generating an array (4) of light spots (S), - irradiating the sample (5) with the array (4) of light spots (S), the light spots (S) interacting with the sample (5) and being imaged on a detector (8), forming an array of image light spots (S'), - scanning the array (4) of light spots (S) into a plurality of positions (Pl, P2, P3) in the sample, - for each position (Pl, P2, P3) of the array (4) of light spots (S) in the sample (5), recording an image of the corresponding array (7) of image light spots (S') on the detector (8), - calculating the relative positions of the array (7) of image light spots (S') on the detector (8) between the recorded images, - calculating the relative positions of the array (4) of light spots (S) in the sample (5) between the recorded images, on the basis of the calculated relative positions of the array (7) of image light spots (S') on the detector (8), and - constructing a complete image of the sample (5) on the basis of the images recorded on the detector (8), the recorded images being attributed to corresponding positions of the array (4) of light spots (S) in the sample (5), these positions taking into account the calculated relative positions of the array (7) of light spots (S) in the sample (5).

Description

Multi-spot scanning optical device for imaging of a sample
Field of the invention
The invention relates to multi-spot scanning imaging, in particular to a multi- spot scanning microscope.
Background of the invention
A scanning microscope is used to image a sample by scanning a focused spot of light (light spot) across the sample and collecting, on a detector, the light interacted with the sample, the interaction being, for instance, reflection, transmission or scattering. The light spot is obtained by focusing a light beam on the sample using a microscope objective.
Multi-spot scanning microscopes have been developed. In a multi-spot scanning microscope, a plurality of light beams are simultaneously irradiated on the sample, forming an array of focused light spots. It is the whole array of light spots that is scanned within the sample, the different focused light spots being spaced apart the ones from the others by a fixed distance and displaced all together. A multi-spot scanning microscope permits to scan (displace) the array of light spots on distances inferior to the dimensions of the sample, since the light spots of the array are distributed on a certain surface of the sample or even on the whole surface of the sample, in which latter case the array only needs to be displaced on the distances corresponding to the surface defined by four proximal light spots in order to cover the whole surface of the sample, all the light spots being displaced together.
At each scanning position, an image of the spots irradiated in the sample is recorded on a detector, which therefore records an array of image light spots. A complete image of the sample is constructed by processing the various recorded images, the processing being performed once all the images have been recorded or in real time; in this latter case the complete image is completed as the images are recorded. Therefore, when the array of light spots is scanned with respect to the detector, the corresponding positions of the focused light spots in the sample have to be determined, in order to build a correct complete image from the images recorded at the different positions.
The term image should be understood in a broad sense. An image comprises information on the different irradiated points of the sample. The information may contain the amount of reflected light at each point, the amount of scattered light at each point, etc. Imaging permits to measure the spatially extended structural properties of the sample. In the description, the term "image" will be used to designate:
- an image, on the detector, of a light spot, which will be called an "image light spot";
- an image, on the detector, of the array of light spots at a particular position, which is recorded in the device and which will be called a "recorded image";
- a complete image of the sample (or of part of the sample), which is constructed on the basis of the recorded images and which will be called a "complete image".
The microscope comprises means for displacing the array of light spots with respect to the sample. Each recorded image corresponds to a particular position of those displacing means and therefore to a corresponding position of the array of light spots within the sample. In order to obtain those recorded images, the displacing means are displaced step by step by predetermined distances and an image is recorded at each predetermined position, or the displacing means are displaced continuously (for instance at a constant speed) and an imaging activator records images at each predetermined position. Each image is attributed to a predetermined position in the sample. However, the displacing means and/or the imaging activator may suffer from lack of accuracy. If the array of light spots is not at the correct predetermined position when the image is taken, the information on the detector will be attributed to incorrect points in the sample, therefore leading to a distorted image.
The actual displacement of the means for displacing the array of light spots should therefore be determined. According to a solution, measuring means may be connected to the means for displacing the array of light spots, so as to measure their actual displacement; however, such a solution is not necessarily very accurate and is not easy to implement. Another solution is to have very accuracy displacing means that can be trusted in; however, such displacing means are expensive. Another solution for determining the position of the array of light spots in the sample consists in performing a measurement of the position of an element of the microscope, which position is related to the relative movement between the array of light spots and the sample. For instance, the chosen element can be the holder of the sample, if this latter is moved during the scanning operation (the array of light spots being fixed); in such a case, the position of a particular array of light spots in the sample is deduced from the position of the holder at the time the particular array was irradiated on the sample. Again, this solution is not very accurate.
Summary of the invention
It is therefore an object of the present invention to provide a method and device for attributing, to the recorded images of an array of light spots on the detector of a multi-spot scanning optical device, the corresponding positions of the array of light spots in the sample.
In accordance with the present invention there is provided a method for imaging a sample with a multi-spot scanning microscope, comprising:
- generating an array of light spots,
- irradiating the sample with the array of light spots, the light spots interacting with the sample and being imaged on a detector, forming an array of image light spots,
- scanning the array of light spots into a plurality of positions in the sample,
- for each position of the array of light spots in the sample, recording an image of the corresponding array of image light spots on the detector,
- calculating the relative positions of the array of image light spots on the detector between the recorded images,
- calculating the relative positions of the array of light spots in the sample between the recorded images, on the basis of the calculated relative positions of the array of image light spots on the detector, and
- constructing a complete image of the sample on the basis of the images recorded on the detector, the recorded images being attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample. Thanks to the invention, the relative positions of the array of image light spots on the detector are calculated and the corresponding relative positions of the array of light spots irradiated in the sample are deduced therefrom. In other words, rather than attributing, to a recorded image, a predetermined position of the light spots in the sample, the invention proposes to attribute, to relative positions of the light spots on the detector between the images, relative positions of the light spots in the sample. With the method of the invention, it is not the absolute position of the array of light spots that is looked for (even if this is possible according to an embodiment of the invention) but the relative position, that is to say, the position relatively to the other positions, in other words, the evolution of the positions between the images. The evolution of the positions depends on the actual positions of the means for scanning the array of light spots in the sample when the images are recorded. Therefore, by calculating the relative positions of the array of light spots in the sample, when reconstructing the complete image of the sample on the basis of the images recorded on the detector, the recorded images are attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample. In other words, the positions - to which the recorded images are attributed - are placed the ones relatively to the others, i.e. the positions are spaced apart the ones from the others, by taking into account their calculated relative positions. This permits to avoid distortions of the complete image; indeed, distortions would occur if the recorded images were attributed to wrong positions of the array of light spots in the sample relatively the ones to the others. The result of the method of the invention is that the proportions of the sample are kept in the complete image that is constructed.
By complete image of the sample, what should be understood is the complete image of a whole object or of part of an object (this part of the object therefore being considered as the sample).
According to an embodiment, the calculated relative positions of the array of light spots in the sample are the same as the corresponding calculated relative positions of the array of image light spots on the detector. According to an embodiment, the relative positions of the array of image light spots on the detector are calculated on the basis of the relative positions of a plurality of light spots of the array that are averaged over the image light spots of the array.
According to an embodiment, the method comprises:
- calculating the position, in at least two successive images on the detector, of at least one image light spot.
According to an embodiment, the relative positions being calculated on the basis of the distances between successive positions, the method comprises:
- calculating the distance between the calculated positions. According to an embodiment, the method comprises:
- calculating the position of at least one image light spot in more than two successive images on the detector and calculating the distance between the calculated positions,
- calculating the ratio between successive calculated distances between successive images and
- multiplying respectively the distances between the positions of the array of light spots in the sample to which the recorded images are attributed when constructing the complete image of the sample, by the calculated ratios.
According to an embodiment, the method comprises:
- calculating, for each image, the position of a plurality of image light spots of the array on the detector and
- calculating a distance ratio averaged over the light spots of the array. According to an embodiment, the optical device comprising a pixelated detector, which comprises a plurality of pixels adapted to be activated by the image light spots, the position of an image light spot is calculated as the position of at least one pixel activated by said image light spot.
According to an embodiment, an image light spot activating a plurality of pixels, the position of the image light spot is calculated as the average position of the activated pixels.
According to an embodiment, the pixels being activated at a certain intensity depending on the intensity of the corresponding image light spot, the average position of the activated pixels is weighted by the respective intensity of their activation. According to an embodiment, the distances on the detector are calibrated, that is to say that, to a distance on the detector, calculated in a number of pixels, an actual distance on the sample is attributed.
According to an embodiment, the relative positions of the array of image light spots are calculated as the average positions of the image light spots on the detector, in arbitrary units.
According to an embodiment, the array of light spots comprising spots aligned in two directions, the positions into which the array of light spots is scanned substantially form a straight line that makes an angle with the directions of the array.
According to an embodiment, the array of light spots comprising spots aligned in two directions, the positions into which the array of light spots is scanned substantially form a set of straight segments, each straight segment having a length substantially equal to the distance between two neighboring light spots in the array and making substantially no angle with one direction of the array, the adjacent straight segments being joined by intermediate segments.
In accordance with the present invention there is also provided a multi-spot scanning microscope comprising:
- means for generating an array of light spots which are intended to irradiate a sample,
- means for imaging the light spots irradiating the sample on a detector, the imaged light spots forming an array of image light spots,
- means for scanning the sample with the array of light spots according to a plurality of positions,
- means for recording, for each position of the array of light spots in the sample, an image of the corresponding array of image light spots on the detector,
- means for calculating the relative positions of the array of image light spots on the detector between the recorded images,
- means for calculating the relative positions of the array of light spots in the sample between the recorded images, on the basis of the calculated relative positions of the array of image light spots on the detector, and
- means for constructing a complete image of the sample on the basis of the images recorded on the detector, the recorded images being attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample.
According to an embodiment, the device comprises a first pixelated detector with high magnification and a second pixelated detector with low magnification.
According to an embodiment, the device comprises means for blurring the image of the image light spots on the detector.
According to an embodiment, the device is a multi-spot scanning microscope.
In accordance with the present invention there is also provided a method for processing a set of recorded images in order to form an image of a sample, each recorded image representing an array of light spots on a detector forming the image of an array of light spots irradiated in the sample, each recorded image corresponding to a different position of the array of lights spots in the sample, the method comprising:
- loading the set of recorded images,
- calculating the relative positions of the array of image light spots on the detector between the recorded images,
- calculating the relative positions of the array of light spots in the sample between the recorded images, on the basis of the calculated relative positions of the array of image light spots on the detector, and
- constructing a complete image of the sample on the basis of the images recorded on the detector, the recorded images being attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample.
Therefore, the images on the detector may be recorded on a suitable medium and the set of recorded images may be provided to data processing means carrying out the above method.
In accordance with the present invention there is also provided a computer program adapted to carry out the above method.
In accordance with the present invention there is also provided a computer program product comprising program code for execution of the above method.
These and other aspects of the invention will be more apparent from the following description with reference to the attached drawings. Brief description of the drawings
- Fig.1 is a schematic diagram of a multi-spot scanning microscope according to the invention;
- Fig.2 shows an image, on the detector of the microscope of Fig.1, of an array of light spots irradiated in the sample;
- Fig.3 shows a zoomed- in image of a single image light spot at a first position on the detector and
- Fig.4 shows a zoomed-in image of the single image light spot of Fig.3 at a second position on the detector.
Detailed description of embodiments
With reference to Fig.1 , an embodiment of an optical scanning microscope 1 according to the invention comprises a light source (not represented), generating a parallel light beam 2 propagating along the optical axis z of the microscope 1. The light beam 2 is incident on a diffractive structure 3, such as a binary phase structure. The diffractive structure 3 generates a multitude of orders of diffraction in numerous directions, the orders interfering and creating at a certain distance an array 4 of light spots S. The light source and the diffractive structure 3 form means for generating an array of light spots 4. Any other suitable means may be contemplated for generating an array of light spots 4.
The focused light spots S of the array 4 are irradiated on a sample 5, which is positioned on a sample support 6, herein comprising for example two glass plates (or slides), as known in the art. The irradiated light spots S form an array 4 in two perpendicular directions, for instance in x and y directions or two perpendicular directions transverse to the direction z of the optical axis and having an angle with the x and y directions. Each light spot is separated from the nearest light spots by a pitch, which is in this embodiment equal in the two directions of the array 4. Of course, other orderings of the light spots within the array 4 may be contemplated, notably more complex configurations.
The irradiated light spots interact with the sample 5 and are imaged on a pixelated detector (sensor) 8 onto which they are focused by imaging optics 9, thereby forming on the detector 8 an array 7 of image light spots S'. The structure of the imaging optics will not be detailed as it is easily accessible to the person skilled in the art. An image light spot S' should be understood as the image, on the detector 8, of a light spot S irradiated in the sample 5.
According to an embodiment, the detector 8 is provided on the same side as the side of incidence of the array of irradiated light spots 4, with appropriate optics for irradiation and detection of the light spots on the same side of the sample 5. According to another embodiment, which is the embodiment represented in Fig.l, the detector 8 is provided on the opposite side, the light passing though the sample 5 and its support 6.
In the embodiment described, each light spot S irradiated on the sample 5 is imaged into a single image light spot S' on the detector 8. Therefore, the array 4 of irradiated light spots S is imaged into an array 7 of image light spots S', the two arrays 4, 7 having a global similar shape, an example of which can be seen on Fig.2 which shows image light spots S' on the detector 8.
Each image light spot S' is analyzed individually to measure a particular feature, for instance its intensity, in order to deduce therefrom a particular characteristic of the point of the sample 5 corresponding to the image light spot S', that is to say, the point of the sample 5 that has been irradiated by the light spot S which, interacted with the sample 5, has been imaged as the image light spot S' on the detector 8. As an example, attenuation in the intensity of an image light spot S' may reveal the presence of an absorbing substance at the corresponding point of the sample 5.
The array of irradiated light spots 4 may be scanned into a plurality of positions in the sample 5. At each position of the array 4 in the sample 5, a corresponding image is recorded on the detector 8; the recorded image comprises the array 7 of image light spots S'. An example of such a recorded image is shown on Fig.2. Each image light spot S' of this array 7 comprises information on the sample 5, as explained above.
A complete image of the sample 5 (or of a certain surface of the sample 5) is constructed on the basis of all the recorded images. Indeed, if the array of irradiated light spots 4 has been scanned into a plurality of positions within all the sample 5 (or within all said certain surface of the sample 5), an image being recorded at each position, the plurality of images contain image light spots S' representing all the positions in the sample 5 (or all the positions in said certain surface of the sample 5). Constructing the complete image may be performed after all the images have been recorded or in real time as the images are recorded.
In order to scan the sample 5, the microscope 1 comprises means for scanning the array 4 of light spots S irradiated in the sample 5. For instance, those means may comprise means for displacing, relatively to the sample 5, the means for generating the array of irradiated light spots 4. According to the embodiment described, the microscope 1 comprises means for displacing the diffractive structure 3 relatively to the sample 5.
In order to scan the sample 5, the array of irradiated light spots 4 is scanned along two directions, for instance the two directions of the array, into a plurality of positions, as explained above; the displacing means may be displaced whether step by step or continuously. In case the dimensions of the array 4 are bigger than the dimensions of the surface of the sample 5 to be scanned, the array 4 only needs to be scanned over the surface of an elementary cell of the array 4, defined by four proximal light spots S, to scan the whole sample surface; indeed, since all the irradiated light spots S are displaced together, if each spot covers the elementary cell, then the array 4 covers the whole scanned surface of the sample 5. In case the dimensions of the array 4 are smaller than the dimensions of the sample 5, the array 4 may be scanned over the surface of the sample 5 that is in the field of view of the imaging optics 9 of the microscope 1 and an image is taken from that surface, and optionally the sample 5 is displaced so as to scan another surface of it.
In any case, the array of irradiated light spots 4 is scanned relatively to the sample 5 in such a manner that each required point of the sample 5 (depending on the required resolution) is touched at least once by a light spot S, in one of the scanning positions. The resolution is determined by the size of the light spots and the distance between two positions of the light spots, that is to say, the distance between two successive positions. For instance, the light spots can be of a diameter equal to lμm, spaced apart in the array by 15μm, the array being moved by lμm between two positions of the array 4 so as to scan the entire surface of the sample 5. The array can also be displaced in such a way that the entire surface of the sample 5 is not scanned but only a discrete portion of it is scanned, for instance if 1 μm diameter light spots are displaced, between two successive positions, by 2μm. According to another embodiment, oversampling is performed, that is to say, the displacement steps between two successive positions of the light spots are smaller than the diameter of the light spots, for instance lμm diameter light spots are displaced, between two successive positions, by 0.5μm; oversampling permits to get a better resolution. According to an embodiment, so-called Nyquist-sampling is performed, where the sampling distance (displacement step) is equal to R/2 for a resolution equal to R, R being determined by the optical cutoff (R=λ/(2*NA) where λ is the wavelength and NA the numerical aperture of the device).
Incidentally, if the two directions of the array 4 make an angle with the translation directions of the means for scanning the array 4, a one direction translation scanning may be sufficient to scan all the positions in the sample 5, for instance if two adjacent light spots are offset, preferably of half the diameter of a light spot (such a configuration leads to an image, on the detector 8, comparable to the one of Fig.2, where it should be contemplated that the array is displaced horizontally between two positions). In other words, the array (4) of light spots (S) comprising spots (S) aligned in two directions, the positions into which the array (4) of light spots (S) is scanned substantially form a straight line that makes an angle with the directions of the array
(4).
According to another embodiment, the array (4) of light spots (S) comprising spots (S) aligned in two directions, the positions into which the array (4) of light spots (S) is scanned substantially form a set of straight segments, each straight segment having a length substantially equal to the distance between two neighboring light spots (S) in the array (4) and making substantially no angle with one direction of the array (4), the adjacent straight segments being joined by intermediate segments; those intermediate segments may be parallel to the second direction of the array.
On Fig.l, one particular irradiated light spot S is represented, in three successive positions Pl, P2, P3 of the array of light spots 4 in the sample 5 (Fig.l is not on scale, for the sake of clarity). From the first position Pl to the second position P2 of the light spot S in the sample 5, the array 4 of irradiated light spots S has been displaced by a first displacement distance dl. From the second position P2 to the third position P3 of the light spot S in the sample 5, the array 4 has been displaced by a second displacement distance d2 in the sample 5.
The light spot S irradiated on the sample 5 is imaged on the detector 8 as an image light spot S'. Each position Pl, P2, P3 of the irradiated light spot S in the sample 5 results in a position P'l, P'2, P'3 of the image light spot S' on the detector 8. From the first position P'l to the second position P'2 on the detector 8, the array 7 of image light spots S' has been displaced by a first displacement distance d'l. From the second position P'2 to the third position P'3 on the detector 8, the array 7 has been displaced by a second displacement distance d'2 on the detector 8.
The means for displacing the diffractive structure 3 may not necessarily be very accurate. Let us consider that the nominal predetermined displacement distance between two successive positions of the array 4 should be 8μm. For instance, between the first and second positions Pl, P2, the means for displacing the array 4 may have displaced the array 4 by a distance dl=8μm, while between the second and third positions P2, P3, it may have displaced the array 4 by a distance d2=10μm, although d2 should theoretically have also been equal to the predetermined distance 8μm. The three successive images on the detector 8, corresponding to said positions Pl, P2, P3, should therefore not be attributed to points in the sample 5 spaced apart by equal distances, otherwise distortions in the constructed complete image will occur.
This is why the method of the invention comprises the following steps:
- for each position of the array of light spots in the sample, recording an image of the corresponding array of image light spots on the detector,
- calculating the relative positions of the array of image light spots on the detector between the recorded images,
- calculating the relative positions of the array of light spots in the sample between the recorded images, on the basis of the calculated relative positions of the array of image light spots, and
- constructing a complete image of the sample on the basis of the images recorded on the detector, the recorded images being attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample. By relative, what should be understood is something that is not absolute, that depends on something. Therefore, a relative position is a position compared to something, that is to say, relatively to something, here, relatively to the other positions.
According to an embodiment, the relative positions are calculated on the basis of the distances between two successive positions.
If the optical device is a multi-spot scanning microscope comprising a pixelated detector 8, which comprises a plurality of pixels that may be activated by the image light spots S', the relative positions of the image light spots S' are calculated as the relative positions of the pixels activated by the corresponding image light spots S'. Indeed, an image light spot S' activates at least one pixel, with a certain intensity.
In the embodiment described, the displacement distances d'l, d'2 between successive positions of the image light spots S' on the detector 8 are used to calculate the relative positions of the array 4 of light spots S in the sample 5.
According to an embodiment, a relative position is calculated as the ratio between successive distances d'l, d'2 between successive positions P'l, P'2, P'3. The relative positions of the array 7 of image light spots S' between successive images are here attributed to the array 4 of irradiated light spots S, that is to say, the relative positions of the array 4 of irradiated light spots S in the sample 5 are set equal to the ratios calculated for the array 7 of image light spots S'.
Indeed, and as already explained above, what is interesting is not the actual values of the distances dl, d2 as such (even if it is possible to calculate those values by further using calibration, as will be explained below). What is interesting is the relation between those distances dl, d2, in the embodiment described, their ratio. Obtaining the ratio between successive distances dl, d2 corresponding to successive images permits, when forming the complete image of the sample 5, to attribute the different recorded images to positions in the sample 5 that are correctly positioned relatively to each other and, thereby, to avoid distortions by keeping the proportions between the distances dl, d2 in the sample. In the complete image, the image light spots S' will be spaced apart by correct distances, the calculated ratios being respected.
Fig.3 and 4 each represent a position P'l, P'2 of a single image light spot S' on the detector 8. Each position P'l, P'2 corresponds to an image recorded on the detector 8. Those positions P'l, P'2 are successive positions, corresponding to successive images. Let us suppose that, between the first and second positions P'l, P'2, there is a displacement distance d'l of 12 pixels. Let us also suppose that between the second and third position P'2, P'3 (not shown on Fig.3 and 4 but shown on Fig.l), there is a displacement distance d'2 equal to 15 pixels. From this distance calculation, it can be concluded that the second displacement distance d'2 is 25% bigger than the first displacement distance d'l. The ratio between the distances d'l, d'2 on the detector 8 is therefore equal to 1.25. This ratio is applied to the corresponding positions Pl, P2, P3 (of the array 4 of irradiated light spots S) that are attributed to the recorded images when constructing the complete image of the sample 5. In other words, in the step of construction of the complete image of the sample 5, the images respectively corresponding to the first, second and third positions P'l, P'2, P'3 on the detector 8 are attributed to positions Pl, P2, P3 in the sample 5 observing the calculated relative positions, that is to say, the calculated ratio of the distances between the positions P'l, P'2, P'3. Therefore, the distance d2 between the second and third positions P2, P3 is 1.25 times the distance dl between the first and second positions Pl, P2. Distortions are hence compensated for.
Imaging can therefore be performed, which measures the spatially extended structural properties of the sample 5.
In the embodiment which has been described, the relative positions are calculated on the basis of the distances between successive positions, between successive images. Other relative positioning can of course be contemplated.
According to an embodiment, the relative positions of the array 7 of image light spots S' are calculated as the averaged positions of the light spots S', in arbitrary units. In other words, the positions of all the image light spots S' are calculated and those positions are averaged in order to give a certain position, which is the average position of the spots. For instance, if each image light spot S' on the detector 8 has coordinates (x,y), the x values of all the image light spots S' are summed and averaged and the y values of all the image light spots S' are summed and averaged, those averaged values giving an average position in arbitrary units. This is done for each recorded image and, from those relative positions in arbitrary units on the detector 8, the relative positions of the array 4 of irradiated light spots S in the sample 5 can be determined. The arbitrary units can in fact be considered as relative coordinates. It should be noticed that, in the calculation of the averaged position in arbitrary units, the image light spots S' situated in the outer bands of the array 7 may not be taken into account if there is a risk that, during the scanning, a spot disappears or appears on a side of the detector between two images; at the contrary, the positions can only be compared between the images if the same spots S', having been displaced, are taken as the basis of the determination of the relative position of the array 7.
According to a particular embodiment of the invention, the means for scanning the array 4 of irradiated light spots S may be cheap ones, since even if those means move the array 4 of light spots S by irregular distances, these irregularities will be compensated for thanks to the method of the invention.
It shall be noted that the method of the invention applies to a relative movement between two assemblies: the means for generating and displacing the array 4 of light spots S on the one hand, the sample 5 and the detector 8 on the other hand. If the sample 5 were moved relatively to the detector 8, the means for generating and displacing the array 4 of light spots S being fixed, the detector 8 would indeed measure information on different positions in the sample 5, but the image light spots S' would appear at the same positions on the detector 8 for every images, therefore forbidding to perform the method of the invention, which establishes a relation between the relative positions on the detector 8 and the relative positions in the sample 5.
As said before, in multi-spot scanning microscopy, if the detector 8 is pixelated, the relative positions of the array 7 of image light spots S' between the images can easily be calculated on the basis of the relative positions of the corresponding activated pixels.
According to an embodiment, the relative positions of the pixels activated by a unique particular image light spot S' between the images are calculated and attributed to the whole array 7 of image light spots S' and, therefore, to the whole array 4 of irradiated light spots S. In other words, the calculations are made for only one image light spot S' between successive images, then considering that the light spots S' of the array 7 of image light spots S' are all displaced by the same distances as this particular light spot S', which are therefore considered as the displacement distance of the array 7. In other words again, it implies considering that all the light spots S' are displaced as a block, without any relative displacements between them. According to another embodiment, the relative positions of a plurality of image light spots S' (or corresponding activated pixels) between images are calculated, those relative positions being averaged. According to a particular embodiment in that case, the relative positions of all the image light spots S' are averaged.
Taking the example of the above mentioned embodiment where the ratio of the displacement distances between successive positions is calculated, the method comprises:
- calculating the position of a plurality of image light spots of the array and
- calculating a ratio averaged over the light spots of the array.
In order to obtain an averaged ratio, three embodiments can be contemplated. Firstly, the positions of the light spots S' are calculated and averaged, then the distances between successive averaged positions are calculated and their ratio is calculated. Secondly, the positions of the light spots S' are calculated, then the distances between successive positions are calculated and averaged and their ratio is calculated. Thirdly, the positions of the light spots S' are calculated, then the distances between successive positions are calculated and their ratio is calculated and then averaged.
In any case, the relative positions of the array 7 are considered as being equal to the averaged relative positions of the image light spots S' over the array 7.
An advantage of averaging the relative positions of the different image light spots S' of the array 7 is that it provides a better accuracy of the position measurement.
In order to calculate the relative positions of the array 7 of image light spots S', the actual positions of the image light spots S' on the detector 8 may be calculated. The total number of pixels N activated per image light spot S' on the detector 8 has to be at least one. According to an embodiment, a plurality of pixels are activated by each image light spot S', so that the position of the image light spots S' can be calculated with sub-pixel accuracy. According to an embodiment, the position of an image light spot S' is calculated as the average position of the pixels activated by the image light spot S', weighted by the respective intensities of the activated pixels. The more pixels per spot, the higher the accuracy of the position determination. According to an embodiment, at least four pixels are activated per image light spot S'. It can be seen on Figs.3 and 4 that each spot activates a plurality of pixels. In such a case, the pixels activated by one image light spot S' are determined according to a predetermined criterion, such as a threshold comparison. Then, in order to take into account the weight of these pixels, an intensity is calculated for each of them, and, according to this, an average position is determined. We may notice that the averaging step applies there to the different pixels activated by a single light spot, the different calculated positions possibly being subsequently averaged over the whole array 7 of image light spots S' for calculating the relative positions of the array 7, as explained above.
Other methods for determining the position of an image light spots S' on the basis of the activated pixels can of course be contemplated such as, for instance, cross- correlation.
Different embodiments for activating more that one pixel per image light spot S' on the detector 8 are proposed.
According to a first embodiment, the multi-spot scanning microscope 1 comprises two detectors (not represented), a first pixelated detector with high magnification and a second pixelated detector with low magnification. A subset of image light spots S' are imaged onto the first detector, which ensures enough pixels per spot for accurate position determination. The complete array of image light spots S' is imaged at the second detector, which allows for a large field. In other words, the second detector is used for global position determination and the first detector with a subset of light spots with many pixels is used for gathering intensity information used for imaging, as well as for determining precise positions of the image light spots. To implement this first embodiment, a beam splitter can be provided that separates the light beams into two paths, one on each detector.
According to a second embodiment, the multi-spot scanning microscope 1 comprises means for blurring the image light spots S' on the detector 8. Those means may for instance comprise defocus means or means for creating spherical aberration, as well known in the art. As a consequence, the image light spot size is enlarged on the detector 8, which activates more pixels and permits to determine the position of the spots more accurately, without lowering the resolution of the microscope. Indeed, since the resolution of an image produced with the multi-spot scanning microscope 1 is determined by the quality of the light spots S irradiated in the sample 5, the blurring in the imaging system does not lower the overall resolution of the system. The maximum blur that is allowed is when two adjacent image light spots S' imaged on the detector 8 can still be separated. Besides, the blurring does not cause any problem for gathering information from the image light spots, since even if the spot intensity is spread over a larger number of pixels, its value is the same; therefore, by integrating the intensity over all the pixels, the value of the intensity is recovered.
According to an embodiment, the distances on the detector 8 are calibrated, that is to say that an actual distance on the sample 5 is attributed to a distance on the detector 8 (calculated in a number of pixels).
Thanks to this calibration, the numerical positions and/or distances on the detector 8 can be calculated in order to give a scale to the constructed image of the sample 5. In such a case, from the positions of all spots S' in an image, an average can be taken to give the position of the array 4 of light spots S in the sample 5 with very high accuracy (sub lOOnm). Therefore, not only the image is not distorted, but also its scale is known.
The method of the invention in particular permits real-time calibration.
Another way of attributing a distance on the detector to a distance in the sample is to know the magnification of the imaging optics.
Another simple way to calibrate the distances on the detector 8 is to use the known distance between two successive light spots of the array 4 of light spots S irradiated on the sample 5. If this distance is known - which is usually the case since the means for generating the array 4 of light spots S are calibrated - therefore the calibration is easily obtained by calculating the number of pixels separating two successive image light spots S' and attributing, to this number of pixels, the known distance.
According to an aspect of the invention, once the array (4) of light spots (S) has been imaged on the detector (8), the images may be recorded on a suitable medium and the set of recorded images may be provided to data processing means, adapted to process the recorded images in order to form a complete image of the sample (5).
A computer program may be provided for enabling the data processing means to carry out the processing method. The computer program may be recorded on a computer program product. While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.
Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid- state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

Claims
1- Method for imaging a sample (5) with a multi-spot scanning microscope, comprising:
- generating an array (4) of light spots (S),
- irradiating the sample (5) with the array (4) of light spots (S), the light spots (S) interacting with the sample (5) and being imaged on a detector (8), forming an array of image light spots (S'),
- scanning the array (4) of light spots (S) into a plurality of positions (Pl, P2, P3) in the sample,
- for each position (Pl, P2, P3) of the array (4) of light spots (S) in the sample (5), recording an image of the corresponding array (7) of image light spots (S') on the detector (8),
- calculating the relative positions of the array (7) of image light spots (S') on the detector (8) between the recorded images,
- calculating the relative positions of the array (4) of light spots (S) in the sample (5) between the recorded images, on the basis of the calculated relative positions of the array (7) of image light spots (S') on the detector (8), and
- constructing a complete image of the sample (5) on the basis of the images recorded on the detector (8), the recorded images being attributed to corresponding positions of the array (4) of light spots (S) in the sample (5), these positions taking into account the calculated relative positions of the array (7) of light spots (S) in the sample (5).
2- Method according to claim 1, wherein the calculated relative positions of the array (4) of light spots (S) in the sample (5) are the same as the corresponding calculated relative positions of the array (7) of image light spots (S') on the detector (8). 3- Method according to claim 1, wherein the relative positions of the array (7) of image light spots (S') on the detector (8) are calculated on the basis of the relative positions of a plurality of light spots (S') of the array (7) that are averaged over the image light spots (S') of the array (7).
4- Method according to claim 1, comprising:
- calculating the position (P' 1, P'2, P'3), in at least two successive images on the detector (8), of at least one image light spot (S').
5- Method according to claim 4 wherein, the relative positions being calculated on the basis of the distances ((dl, d2), (d'l, d'2)) between successive positions ((Pl, P2, P3), (P' 1, P'2, P'3)), the method comprises:
- calculating the distance (d'l, d'2) between the calculated positions (P' 1, P'2, P'3).
6- Method according to claim 4 comprising:
- calculating the position (P' 1, P'2, P'3) of at least one image light spot (S') in more than two successive images on the detector (8) and calculating the distance (d'l, d'2) between the calculated positions (P' 1, P'2, P'3),
- calculating the ratio between successive calculated distances (d'l, d'2) between successive images and
- multiplying respectively the distances (dl, d2) between the positions (Pl, P2, P3) of the array (4) of light spots (S) in the sample (5) to which the recorded images are attributed when constructing the complete image of the sample (5), by the calculated ratios.
7- Method according to claim 6 comprising:
- calculating, for each image, the position of a plurality of image light spots (S') of the array (7) on the detector (8) and
- calculating a distance ratio averaged over the light spots (S') of the array (7). 8- Method according to claim 4 wherein, the optical device comprising a pixelated detector (8) which comprises a plurality of pixels adapted to be activated by the image light spots (S'), the position of an image light spot (S') is calculated as the position of at least one pixel activated by said image light spot (S').
9- Method according to claim 8 wherein, an image light spot (S') activating a plurality of pixels, the position (P' 1, P'2, P'3) of the image light spot (S') is calculated as the average position of the activated pixels.
10- Method according to claim 9 wherein, the pixels being activated at a certain intensity depending on the intensity of the corresponding image light spot (S'), the average position of the activated pixels is weighted by the respective intensity of their activation.
11- Method according to claim 4, wherein the relative positions of the array (7) of image light spots (S') are calculated as the average positions of the image light spots (S') on the detector (8), in arbitrary units.
12- Multi-spot scanning microscope comprising:
- means (2, 3) for generating an array (4) of light spots (S) which are intended to irradiate a sample (5)
- means (9) for imaging the light spots irradiating the sample (5) on a detector (8), the imaged light spots (S') forming an array (7) of image light spots (S'),
- means for scanning the sample with the array (4) of light spots (S) according to a plurality of positions (Pl, P2, P3),
- means for recording, for each position (Pl, P2, P3) of the array (4) of light spots (S) in the sample (5), an image of the corresponding array (7) of image light spots (S') on the detector (8),
- means for calculating the relative positions of the array (7) of image light spots (S') on the detector (8) between the recorded images, - means for calculating the relative positions of the array (4) of light spots (S) in the sample (5) between the recorded images, on the basis of the calculated relative positions of the array (7) of image light spots (S') on the detector (8), and
- means for constructing a complete image of the sample (5) on the basis of the images recorded on the detector (8), the recorded images being attributed to corresponding positions of the array (4) of light spots (S) in the sample (5), these positions taking into account the calculated relative positions of the array (4) of light spots (S) in the sample (5).
13- Device according to claim 12, comprising a first pixelated detector with high magnification and a second pixelated detector with low magnification.
14- Device according to claim 12, comprising means for blurring the image of the image light spots (S') on the detector (8).
15- Method for processing a set of recorded images in order to form an image of a sample (5), each recorded image representing an array (7) of light spots (S') on a detector (8) forming the image of an array (4) of light spots (S) irradiated in the sample (5), each recorded image corresponding to a different position (Pl, P2, P3) of the array (4) of lights spots (S) in the sample (5), the method comprising:
- loading the set of recorded images,
- calculating the relative positions of the array (7) of image light spots (S') on the detector (8) between the recorded images,
- calculating the relative positions of the array (4) of light spots (S) in the sample (5) between the recorded images, on the basis of the calculated relative positions of the array (7) of image light spots (S') on the detector (8), and
- constructing a complete image of the sample (5) on the basis of the images recorded on the detector (8), the recorded images being attributed to corresponding positions of the array (4) of light spots (S) in the sample (5), these positions taking into account the calculated relative positions of the array (7) of light spots (S) in the sample (5). 16- Computer program adapted to carry out the method of claim 15.
17- Computer program product comprising program code for execution of the method of claim 15.
PCT/IB2008/052338 2007-06-15 2008-06-13 Multi-spot scanning optical device for imaging of a sample WO2008152605A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP07301112 2007-06-15
EP07301112.4 2007-06-15

Publications (1)

Publication Number Publication Date
WO2008152605A1 true WO2008152605A1 (en) 2008-12-18

Family

ID=39713724

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2008/052338 WO2008152605A1 (en) 2007-06-15 2008-06-13 Multi-spot scanning optical device for imaging of a sample

Country Status (1)

Country Link
WO (1) WO2008152605A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013049646A1 (en) 2011-09-29 2013-04-04 Fei Company Microscope device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4769698A (en) * 1985-10-04 1988-09-06 National Biomedical Research Foundation Interactive microscopic image display system and method
US4806004A (en) * 1987-07-10 1989-02-21 California Institute Of Technology Scanning microscopy
US5239178A (en) * 1990-11-10 1993-08-24 Carl Zeiss Optical device with an illuminating grid and detector grid arranged confocally to an object
US20020090127A1 (en) * 2001-01-11 2002-07-11 Interscope Technologies, Inc. System for creating microscopic digital montage images
US6697164B1 (en) * 1998-08-05 2004-02-24 Cadent Ltd. Imaging a three-dimensional structure by confocal focussing an array of light beams
EP1510851A1 (en) * 2003-08-25 2005-03-02 Cadent Limited Apparatus and method for providing high intensity non-coherent light and for speckle reduction
US20060072191A1 (en) * 2004-10-06 2006-04-06 Yokogawa Electric Corporation Confocal microscope

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4769698A (en) * 1985-10-04 1988-09-06 National Biomedical Research Foundation Interactive microscopic image display system and method
US4806004A (en) * 1987-07-10 1989-02-21 California Institute Of Technology Scanning microscopy
US5239178A (en) * 1990-11-10 1993-08-24 Carl Zeiss Optical device with an illuminating grid and detector grid arranged confocally to an object
US6697164B1 (en) * 1998-08-05 2004-02-24 Cadent Ltd. Imaging a three-dimensional structure by confocal focussing an array of light beams
US20020090127A1 (en) * 2001-01-11 2002-07-11 Interscope Technologies, Inc. System for creating microscopic digital montage images
EP1510851A1 (en) * 2003-08-25 2005-03-02 Cadent Limited Apparatus and method for providing high intensity non-coherent light and for speckle reduction
US20060072191A1 (en) * 2004-10-06 2006-04-06 Yokogawa Electric Corporation Confocal microscope

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013049646A1 (en) 2011-09-29 2013-04-04 Fei Company Microscope device
CN103890633A (en) * 2011-09-29 2014-06-25 Fei公司 Microscope device
EP2761356A1 (en) * 2011-09-29 2014-08-06 FEI Company Microscope device
JP2014532197A (en) * 2011-09-29 2014-12-04 エフ・イ−・アイ・カンパニー Microscope device
EP2761356A4 (en) * 2011-09-29 2015-02-25 Fei Co Microscope device
US9606345B2 (en) 2011-09-29 2017-03-28 Fei Company Microscope device

Similar Documents

Publication Publication Date Title
JP6934856B2 (en) Optical sheet microscope that simultaneously images multiple target surfaces
US8731272B2 (en) Computational adaptive optics for interferometric synthetic aperture microscopy and other interferometric imaging
JP4943946B2 (en) Eccentricity measuring device
JP4747243B2 (en) Method and apparatus for optical grasping of samples by optical deep decomposition
EP2452221A1 (en) Microscopy with adaptive optics
US20110025837A1 (en) method of imaging a sample
US11243390B2 (en) Apparatus and method of providing parameter estimation
KR20160016962A (en) Adjustment method for adaptive optics system, adaptive optics system, and storage medium storing program for adaptive optics system
JP2011508255A (en) Scanning microscope and sample imaging method
JP2011530708A (en) Measurement and correction of lens distortion in a multi-spot scanner
JP2004219987A (en) Multilayer observation type optical microscope and multilayer observation unit
Luo et al. Area scanning method for 3D surface profilometry based on an adaptive confocal microscope
CN105758381A (en) Method for detecting inclination of camera die set based on frequency spectrum analysis
US20230393379A1 (en) Apparatus and method of estimating values from images
JPH1068616A (en) Shape measuring equipment
WO2008152605A1 (en) Multi-spot scanning optical device for imaging of a sample
JP2016515718A (en) Optical microscopy for the localization of point objects
CN114076670B (en) Splicing primary mirror common-phase error detection method, system and storage medium
US11119299B2 (en) Area scanning confocal microscopy (ASCM)
DE10049296B4 (en) Optical arrangement and method for deflecting a light beam
US20230069794A1 (en) Dual-mode restoration microscopy
CN114371549A (en) Quantitative phase imaging method and system based on multi-focus multiplexing lens

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08763325

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 08763325

Country of ref document: EP

Kind code of ref document: A1