WO2008152605A1 - Multi-spot scanning optical device for imaging of a sample - Google Patents
Multi-spot scanning optical device for imaging of a sample Download PDFInfo
- Publication number
- WO2008152605A1 WO2008152605A1 PCT/IB2008/052338 IB2008052338W WO2008152605A1 WO 2008152605 A1 WO2008152605 A1 WO 2008152605A1 IB 2008052338 W IB2008052338 W IB 2008052338W WO 2008152605 A1 WO2008152605 A1 WO 2008152605A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- array
- light spots
- sample
- detector
- image
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/0004—Microscopes specially adapted for specific applications
- G02B21/002—Scanning microscopes
- G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
- G02B21/0032—Optical details of illumination, e.g. light-sources, pinholes, beam splitters, slits, fibers
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/0004—Microscopes specially adapted for specific applications
- G02B21/002—Scanning microscopes
- G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
- G02B21/008—Details of detection or image processing, including general computer control
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/365—Control or image processing arrangements for digital or video microscopes
- G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
Definitions
- Multi-spot scanning optical device for imaging of a sample
- the invention relates to multi-spot scanning imaging, in particular to a multi- spot scanning microscope.
- a scanning microscope is used to image a sample by scanning a focused spot of light (light spot) across the sample and collecting, on a detector, the light interacted with the sample, the interaction being, for instance, reflection, transmission or scattering.
- the light spot is obtained by focusing a light beam on the sample using a microscope objective.
- Multi-spot scanning microscopes have been developed.
- a multi-spot scanning microscope a plurality of light beams are simultaneously irradiated on the sample, forming an array of focused light spots. It is the whole array of light spots that is scanned within the sample, the different focused light spots being spaced apart the ones from the others by a fixed distance and displaced all together.
- a multi-spot scanning microscope permits to scan (displace) the array of light spots on distances inferior to the dimensions of the sample, since the light spots of the array are distributed on a certain surface of the sample or even on the whole surface of the sample, in which latter case the array only needs to be displaced on the distances corresponding to the surface defined by four proximal light spots in order to cover the whole surface of the sample, all the light spots being displaced together.
- an image of the spots irradiated in the sample is recorded on a detector, which therefore records an array of image light spots.
- a complete image of the sample is constructed by processing the various recorded images, the processing being performed once all the images have been recorded or in real time; in this latter case the complete image is completed as the images are recorded. Therefore, when the array of light spots is scanned with respect to the detector, the corresponding positions of the focused light spots in the sample have to be determined, in order to build a correct complete image from the images recorded at the different positions.
- image should be understood in a broad sense.
- An image comprises information on the different irradiated points of the sample. The information may contain the amount of reflected light at each point, the amount of scattered light at each point, etc. Imaging permits to measure the spatially extended structural properties of the sample.
- image will be used to designate:
- the microscope comprises means for displacing the array of light spots with respect to the sample.
- Each recorded image corresponds to a particular position of those displacing means and therefore to a corresponding position of the array of light spots within the sample.
- the displacing means are displaced step by step by predetermined distances and an image is recorded at each predetermined position, or the displacing means are displaced continuously (for instance at a constant speed) and an imaging activator records images at each predetermined position.
- Each image is attributed to a predetermined position in the sample.
- the displacing means and/or the imaging activator may suffer from lack of accuracy. If the array of light spots is not at the correct predetermined position when the image is taken, the information on the detector will be attributed to incorrect points in the sample, therefore leading to a distorted image.
- measuring means may be connected to the means for displacing the array of light spots, so as to measure their actual displacement; however, such a solution is not necessarily very accurate and is not easy to implement.
- Another solution is to have very accuracy displacing means that can be trusted in; however, such displacing means are expensive.
- Another solution for determining the position of the array of light spots in the sample consists in performing a measurement of the position of an element of the microscope, which position is related to the relative movement between the array of light spots and the sample.
- the chosen element can be the holder of the sample, if this latter is moved during the scanning operation (the array of light spots being fixed); in such a case, the position of a particular array of light spots in the sample is deduced from the position of the holder at the time the particular array was irradiated on the sample. Again, this solution is not very accurate.
- a method for imaging a sample with a multi-spot scanning microscope comprising:
- the invention proposes to attribute, to relative positions of the light spots on the detector between the images, relative positions of the light spots in the sample.
- the method of the invention it is not the absolute position of the array of light spots that is looked for (even if this is possible according to an embodiment of the invention) but the relative position, that is to say, the position relatively to the other positions, in other words, the evolution of the positions between the images.
- the evolution of the positions depends on the actual positions of the means for scanning the array of light spots in the sample when the images are recorded. Therefore, by calculating the relative positions of the array of light spots in the sample, when reconstructing the complete image of the sample on the basis of the images recorded on the detector, the recorded images are attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample.
- the positions - to which the recorded images are attributed - are placed the ones relatively to the others, i.e. the positions are spaced apart the ones from the others, by taking into account their calculated relative positions.
- This permits to avoid distortions of the complete image; indeed, distortions would occur if the recorded images were attributed to wrong positions of the array of light spots in the sample relatively the ones to the others.
- the result of the method of the invention is that the proportions of the sample are kept in the complete image that is constructed.
- the calculated relative positions of the array of light spots in the sample are the same as the corresponding calculated relative positions of the array of image light spots on the detector.
- the relative positions of the array of image light spots on the detector are calculated on the basis of the relative positions of a plurality of light spots of the array that are averaged over the image light spots of the array.
- the method comprises:
- the method comprises:
- the method comprises:
- the method comprises:
- the optical device comprising a pixelated detector, which comprises a plurality of pixels adapted to be activated by the image light spots, the position of an image light spot is calculated as the position of at least one pixel activated by said image light spot.
- an image light spot activating a plurality of pixels the position of the image light spot is calculated as the average position of the activated pixels.
- the pixels being activated at a certain intensity depending on the intensity of the corresponding image light spot the average position of the activated pixels is weighted by the respective intensity of their activation.
- the distances on the detector are calibrated, that is to say that, to a distance on the detector, calculated in a number of pixels, an actual distance on the sample is attributed.
- the relative positions of the array of image light spots are calculated as the average positions of the image light spots on the detector, in arbitrary units.
- the array of light spots comprising spots aligned in two directions, the positions into which the array of light spots is scanned substantially form a straight line that makes an angle with the directions of the array.
- the array of light spots comprising spots aligned in two directions, the positions into which the array of light spots is scanned substantially form a set of straight segments, each straight segment having a length substantially equal to the distance between two neighboring light spots in the array and making substantially no angle with one direction of the array, the adjacent straight segments being joined by intermediate segments.
- a multi-spot scanning microscope comprising:
- the device comprises a first pixelated detector with high magnification and a second pixelated detector with low magnification.
- the device comprises means for blurring the image of the image light spots on the detector.
- the device is a multi-spot scanning microscope.
- a method for processing a set of recorded images in order to form an image of a sample each recorded image representing an array of light spots on a detector forming the image of an array of light spots irradiated in the sample, each recorded image corresponding to a different position of the array of lights spots in the sample, the method comprising:
- the images on the detector may be recorded on a suitable medium and the set of recorded images may be provided to data processing means carrying out the above method.
- - Fig.1 is a schematic diagram of a multi-spot scanning microscope according to the invention.
- - Fig.2 shows an image, on the detector of the microscope of Fig.1, of an array of light spots irradiated in the sample;
- - Fig.3 shows a zoomed- in image of a single image light spot at a first position on the detector
- - Fig.4 shows a zoomed-in image of the single image light spot of Fig.3 at a second position on the detector.
- an embodiment of an optical scanning microscope 1 comprises a light source (not represented), generating a parallel light beam 2 propagating along the optical axis z of the microscope 1.
- the light beam 2 is incident on a diffractive structure 3, such as a binary phase structure.
- the diffractive structure 3 generates a multitude of orders of diffraction in numerous directions, the orders interfering and creating at a certain distance an array 4 of light spots S.
- the light source and the diffractive structure 3 form means for generating an array of light spots 4. Any other suitable means may be contemplated for generating an array of light spots 4.
- the focused light spots S of the array 4 are irradiated on a sample 5, which is positioned on a sample support 6, herein comprising for example two glass plates (or slides), as known in the art.
- the irradiated light spots S form an array 4 in two perpendicular directions, for instance in x and y directions or two perpendicular directions transverse to the direction z of the optical axis and having an angle with the x and y directions.
- Each light spot is separated from the nearest light spots by a pitch, which is in this embodiment equal in the two directions of the array 4.
- other orderings of the light spots within the array 4 may be contemplated, notably more complex configurations.
- the irradiated light spots interact with the sample 5 and are imaged on a pixelated detector (sensor) 8 onto which they are focused by imaging optics 9, thereby forming on the detector 8 an array 7 of image light spots S'.
- the structure of the imaging optics will not be detailed as it is easily accessible to the person skilled in the art.
- An image light spot S' should be understood as the image, on the detector 8, of a light spot S irradiated in the sample 5.
- the detector 8 is provided on the same side as the side of incidence of the array of irradiated light spots 4, with appropriate optics for irradiation and detection of the light spots on the same side of the sample 5.
- the detector 8 is provided on the opposite side, the light passing though the sample 5 and its support 6.
- each light spot S irradiated on the sample 5 is imaged into a single image light spot S' on the detector 8. Therefore, the array 4 of irradiated light spots S is imaged into an array 7 of image light spots S', the two arrays 4, 7 having a global similar shape, an example of which can be seen on Fig.2 which shows image light spots S' on the detector 8.
- Each image light spot S' is analyzed individually to measure a particular feature, for instance its intensity, in order to deduce therefrom a particular characteristic of the point of the sample 5 corresponding to the image light spot S', that is to say, the point of the sample 5 that has been irradiated by the light spot S which, interacted with the sample 5, has been imaged as the image light spot S' on the detector 8.
- a particular characteristic of the point of the sample 5 corresponding to the image light spot S' that is to say, the point of the sample 5 that has been irradiated by the light spot S which, interacted with the sample 5, has been imaged as the image light spot S' on the detector 8.
- attenuation in the intensity of an image light spot S' may reveal the presence of an absorbing substance at the corresponding point of the sample 5.
- the array of irradiated light spots 4 may be scanned into a plurality of positions in the sample 5. At each position of the array 4 in the sample 5, a corresponding image is recorded on the detector 8; the recorded image comprises the array 7 of image light spots S'. An example of such a recorded image is shown on Fig.2. Each image light spot S' of this array 7 comprises information on the sample 5, as explained above.
- a complete image of the sample 5 (or of a certain surface of the sample 5) is constructed on the basis of all the recorded images. Indeed, if the array of irradiated light spots 4 has been scanned into a plurality of positions within all the sample 5 (or within all said certain surface of the sample 5), an image being recorded at each position, the plurality of images contain image light spots S' representing all the positions in the sample 5 (or all the positions in said certain surface of the sample 5). Constructing the complete image may be performed after all the images have been recorded or in real time as the images are recorded.
- the microscope 1 comprises means for scanning the array 4 of light spots S irradiated in the sample 5.
- those means may comprise means for displacing, relatively to the sample 5, the means for generating the array of irradiated light spots 4.
- the microscope 1 comprises means for displacing the diffractive structure 3 relatively to the sample 5.
- the array of irradiated light spots 4 is scanned along two directions, for instance the two directions of the array, into a plurality of positions, as explained above; the displacing means may be displaced whether step by step or continuously.
- the array 4 only needs to be scanned over the surface of an elementary cell of the array 4, defined by four proximal light spots S, to scan the whole sample surface; indeed, since all the irradiated light spots S are displaced together, if each spot covers the elementary cell, then the array 4 covers the whole scanned surface of the sample 5.
- the array 4 may be scanned over the surface of the sample 5 that is in the field of view of the imaging optics 9 of the microscope 1 and an image is taken from that surface, and optionally the sample 5 is displaced so as to scan another surface of it.
- the array of irradiated light spots 4 is scanned relatively to the sample 5 in such a manner that each required point of the sample 5 (depending on the required resolution) is touched at least once by a light spot S, in one of the scanning positions.
- the resolution is determined by the size of the light spots and the distance between two positions of the light spots, that is to say, the distance between two successive positions.
- the light spots can be of a diameter equal to l ⁇ m, spaced apart in the array by 15 ⁇ m, the array being moved by l ⁇ m between two positions of the array 4 so as to scan the entire surface of the sample 5.
- the array can also be displaced in such a way that the entire surface of the sample 5 is not scanned but only a discrete portion of it is scanned, for instance if 1 ⁇ m diameter light spots are displaced, between two successive positions, by 2 ⁇ m.
- oversampling is performed, that is to say, the displacement steps between two successive positions of the light spots are smaller than the diameter of the light spots, for instance l ⁇ m diameter light spots are displaced, between two successive positions, by 0.5 ⁇ m; oversampling permits to get a better resolution.
- a one direction translation scanning may be sufficient to scan all the positions in the sample 5, for instance if two adjacent light spots are offset, preferably of half the diameter of a light spot (such a configuration leads to an image, on the detector 8, comparable to the one of Fig.2, where it should be contemplated that the array is displaced horizontally between two positions).
- the array (4) of light spots (S) comprising spots (S) aligned in two directions, the positions into which the array (4) of light spots (S) is scanned substantially form a straight line that makes an angle with the directions of the array
- the array (4) of light spots (S) comprising spots (S) aligned in two directions, the positions into which the array (4) of light spots (S) is scanned substantially form a set of straight segments, each straight segment having a length substantially equal to the distance between two neighboring light spots (S) in the array (4) and making substantially no angle with one direction of the array (4), the adjacent straight segments being joined by intermediate segments; those intermediate segments may be parallel to the second direction of the array.
- Fig.l one particular irradiated light spot S is represented, in three successive positions Pl, P2, P3 of the array of light spots 4 in the sample 5 (Fig.l is not on scale, for the sake of clarity). From the first position Pl to the second position P2 of the light spot S in the sample 5, the array 4 of irradiated light spots S has been displaced by a first displacement distance dl. From the second position P2 to the third position P3 of the light spot S in the sample 5, the array 4 has been displaced by a second displacement distance d2 in the sample 5.
- the light spot S irradiated on the sample 5 is imaged on the detector 8 as an image light spot S'.
- Each position Pl, P2, P3 of the irradiated light spot S in the sample 5 results in a position P'l, P'2, P'3 of the image light spot S' on the detector 8.
- the array 7 of image light spots S' has been displaced by a first displacement distance d'l.
- the array 7 has been displaced by a second displacement distance d'2 on the detector 8.
- the three successive images on the detector 8, corresponding to said positions Pl, P2, P3, should therefore not be attributed to points in the sample 5 spaced apart by equal distances, otherwise distortions in the constructed complete image will occur.
- a relative position is a position compared to something, that is to say, relatively to something, here, relatively to the other positions.
- the relative positions are calculated on the basis of the distances between two successive positions.
- the optical device is a multi-spot scanning microscope comprising a pixelated detector 8, which comprises a plurality of pixels that may be activated by the image light spots S'
- the relative positions of the image light spots S' are calculated as the relative positions of the pixels activated by the corresponding image light spots S'. Indeed, an image light spot S' activates at least one pixel, with a certain intensity.
- the displacement distances d'l, d'2 between successive positions of the image light spots S' on the detector 8 are used to calculate the relative positions of the array 4 of light spots S in the sample 5.
- a relative position is calculated as the ratio between successive distances d'l, d'2 between successive positions P'l, P'2, P'3.
- the relative positions of the array 7 of image light spots S' between successive images are here attributed to the array 4 of irradiated light spots S, that is to say, the relative positions of the array 4 of irradiated light spots S in the sample 5 are set equal to the ratios calculated for the array 7 of image light spots S'.
- Fig.3 and 4 each represent a position P'l, P'2 of a single image light spot S' on the detector 8. Each position P'l, P'2 corresponds to an image recorded on the detector 8. Those positions P'l, P'2 are successive positions, corresponding to successive images. Let us suppose that, between the first and second positions P'l, P'2, there is a displacement distance d'l of 12 pixels. Let us also suppose that between the second and third position P'2, P'3 (not shown on Fig.3 and 4 but shown on Fig.l), there is a displacement distance d'2 equal to 15 pixels. From this distance calculation, it can be concluded that the second displacement distance d'2 is 25% bigger than the first displacement distance d'l.
- the ratio between the distances d'l, d'2 on the detector 8 is therefore equal to 1.25.
- This ratio is applied to the corresponding positions Pl, P2, P3 (of the array 4 of irradiated light spots S) that are attributed to the recorded images when constructing the complete image of the sample 5.
- the images respectively corresponding to the first, second and third positions P'l, P'2, P'3 on the detector 8 are attributed to positions Pl, P2, P3 in the sample 5 observing the calculated relative positions, that is to say, the calculated ratio of the distances between the positions P'l, P'2, P'3. Therefore, the distance d2 between the second and third positions P2, P3 is 1.25 times the distance dl between the first and second positions Pl, P2. Distortions are hence compensated for.
- Imaging can therefore be performed, which measures the spatially extended structural properties of the sample 5.
- the relative positions are calculated on the basis of the distances between successive positions, between successive images. Other relative positioning can of course be contemplated.
- the relative positions of the array 7 of image light spots S' are calculated as the averaged positions of the light spots S', in arbitrary units.
- the positions of all the image light spots S' are calculated and those positions are averaged in order to give a certain position, which is the average position of the spots. For instance, if each image light spot S' on the detector 8 has coordinates (x,y), the x values of all the image light spots S' are summed and averaged and the y values of all the image light spots S' are summed and averaged, those averaged values giving an average position in arbitrary units.
- the relative positions of the array 4 of irradiated light spots S in the sample 5 can be determined.
- the arbitrary units can in fact be considered as relative coordinates. It should be noticed that, in the calculation of the averaged position in arbitrary units, the image light spots S' situated in the outer bands of the array 7 may not be taken into account if there is a risk that, during the scanning, a spot disappears or appears on a side of the detector between two images; at the contrary, the positions can only be compared between the images if the same spots S', having been displaced, are taken as the basis of the determination of the relative position of the array 7.
- the means for scanning the array 4 of irradiated light spots S may be cheap ones, since even if those means move the array 4 of light spots S by irregular distances, these irregularities will be compensated for thanks to the method of the invention.
- the method of the invention applies to a relative movement between two assemblies: the means for generating and displacing the array 4 of light spots S on the one hand, the sample 5 and the detector 8 on the other hand. If the sample 5 were moved relatively to the detector 8, the means for generating and displacing the array 4 of light spots S being fixed, the detector 8 would indeed measure information on different positions in the sample 5, but the image light spots S' would appear at the same positions on the detector 8 for every images, therefore forbidding to perform the method of the invention, which establishes a relation between the relative positions on the detector 8 and the relative positions in the sample 5.
- the relative positions of the array 7 of image light spots S' between the images can easily be calculated on the basis of the relative positions of the corresponding activated pixels.
- the relative positions of the pixels activated by a unique particular image light spot S' between the images are calculated and attributed to the whole array 7 of image light spots S' and, therefore, to the whole array 4 of irradiated light spots S.
- the calculations are made for only one image light spot S' between successive images, then considering that the light spots S' of the array 7 of image light spots S' are all displaced by the same distances as this particular light spot S', which are therefore considered as the displacement distance of the array 7.
- the relative positions of a plurality of image light spots S' (or corresponding activated pixels) between images are calculated, those relative positions being averaged. According to a particular embodiment in that case, the relative positions of all the image light spots S' are averaged.
- the method comprises:
- three embodiments can be contemplated. Firstly, the positions of the light spots S' are calculated and averaged, then the distances between successive averaged positions are calculated and their ratio is calculated. Secondly, the positions of the light spots S' are calculated, then the distances between successive positions are calculated and averaged and their ratio is calculated. Thirdly, the positions of the light spots S' are calculated, then the distances between successive positions are calculated and their ratio is calculated and then averaged.
- the relative positions of the array 7 are considered as being equal to the averaged relative positions of the image light spots S' over the array 7.
- An advantage of averaging the relative positions of the different image light spots S' of the array 7 is that it provides a better accuracy of the position measurement.
- the actual positions of the image light spots S' on the detector 8 may be calculated.
- the total number of pixels N activated per image light spot S' on the detector 8 has to be at least one.
- a plurality of pixels are activated by each image light spot S', so that the position of the image light spots S' can be calculated with sub-pixel accuracy.
- the position of an image light spot S' is calculated as the average position of the pixels activated by the image light spot S', weighted by the respective intensities of the activated pixels. The more pixels per spot, the higher the accuracy of the position determination.
- at least four pixels are activated per image light spot S'.
- each spot activates a plurality of pixels.
- the pixels activated by one image light spot S' are determined according to a predetermined criterion, such as a threshold comparison. Then, in order to take into account the weight of these pixels, an intensity is calculated for each of them, and, according to this, an average position is determined.
- averaging step applies there to the different pixels activated by a single light spot, the different calculated positions possibly being subsequently averaged over the whole array 7 of image light spots S' for calculating the relative positions of the array 7, as explained above.
- the multi-spot scanning microscope 1 comprises two detectors (not represented), a first pixelated detector with high magnification and a second pixelated detector with low magnification.
- a subset of image light spots S' are imaged onto the first detector, which ensures enough pixels per spot for accurate position determination.
- the complete array of image light spots S' is imaged at the second detector, which allows for a large field.
- the second detector is used for global position determination and the first detector with a subset of light spots with many pixels is used for gathering intensity information used for imaging, as well as for determining precise positions of the image light spots.
- a beam splitter can be provided that separates the light beams into two paths, one on each detector.
- the multi-spot scanning microscope 1 comprises means for blurring the image light spots S' on the detector 8.
- Those means may for instance comprise defocus means or means for creating spherical aberration, as well known in the art.
- the image light spot size is enlarged on the detector 8, which activates more pixels and permits to determine the position of the spots more accurately, without lowering the resolution of the microscope.
- the resolution of an image produced with the multi-spot scanning microscope 1 is determined by the quality of the light spots S irradiated in the sample 5, the blurring in the imaging system does not lower the overall resolution of the system.
- the maximum blur that is allowed is when two adjacent image light spots S' imaged on the detector 8 can still be separated.
- the blurring does not cause any problem for gathering information from the image light spots, since even if the spot intensity is spread over a larger number of pixels, its value is the same; therefore, by integrating the intensity over all the pixels, the value of the intensity is recovered.
- the distances on the detector 8 are calibrated, that is to say that an actual distance on the sample 5 is attributed to a distance on the detector 8 (calculated in a number of pixels).
- the numerical positions and/or distances on the detector 8 can be calculated in order to give a scale to the constructed image of the sample 5.
- an average can be taken to give the position of the array 4 of light spots S in the sample 5 with very high accuracy (sub lOOnm). Therefore, not only the image is not distorted, but also its scale is known.
- the method of the invention in particular permits real-time calibration.
- Another way of attributing a distance on the detector to a distance in the sample is to know the magnification of the imaging optics.
- Another simple way to calibrate the distances on the detector 8 is to use the known distance between two successive light spots of the array 4 of light spots S irradiated on the sample 5. If this distance is known - which is usually the case since the means for generating the array 4 of light spots S are calibrated - therefore the calibration is easily obtained by calculating the number of pixels separating two successive image light spots S' and attributing, to this number of pixels, the known distance.
- the images may be recorded on a suitable medium and the set of recorded images may be provided to data processing means, adapted to process the recorded images in order to form a complete image of the sample (5).
- a computer program may be provided for enabling the data processing means to carry out the processing method.
- the computer program may be recorded on a computer program product. While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.
- a computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid- state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.
Abstract
The invention relates to a method for imaging a sample (5) with a multi-spot scanning microscope, comprising: - generating an array (4) of light spots (S), - irradiating the sample (5) with the array (4) of light spots (S), the light spots (S) interacting with the sample (5) and being imaged on a detector (8), forming an array of image light spots (S'), - scanning the array (4) of light spots (S) into a plurality of positions (Pl, P2, P3) in the sample, - for each position (Pl, P2, P3) of the array (4) of light spots (S) in the sample (5), recording an image of the corresponding array (7) of image light spots (S') on the detector (8), - calculating the relative positions of the array (7) of image light spots (S') on the detector (8) between the recorded images, - calculating the relative positions of the array (4) of light spots (S) in the sample (5) between the recorded images, on the basis of the calculated relative positions of the array (7) of image light spots (S') on the detector (8), and - constructing a complete image of the sample (5) on the basis of the images recorded on the detector (8), the recorded images being attributed to corresponding positions of the array (4) of light spots (S) in the sample (5), these positions taking into account the calculated relative positions of the array (7) of light spots (S) in the sample (5).
Description
Multi-spot scanning optical device for imaging of a sample
Field of the invention
The invention relates to multi-spot scanning imaging, in particular to a multi- spot scanning microscope.
Background of the invention
A scanning microscope is used to image a sample by scanning a focused spot of light (light spot) across the sample and collecting, on a detector, the light interacted with the sample, the interaction being, for instance, reflection, transmission or scattering. The light spot is obtained by focusing a light beam on the sample using a microscope objective.
Multi-spot scanning microscopes have been developed. In a multi-spot scanning microscope, a plurality of light beams are simultaneously irradiated on the sample, forming an array of focused light spots. It is the whole array of light spots that is scanned within the sample, the different focused light spots being spaced apart the ones from the others by a fixed distance and displaced all together. A multi-spot scanning microscope permits to scan (displace) the array of light spots on distances inferior to the dimensions of the sample, since the light spots of the array are distributed on a certain surface of the sample or even on the whole surface of the sample, in which latter case the array only needs to be displaced on the distances corresponding to the surface defined by four proximal light spots in order to cover the whole surface of the sample, all the light spots being displaced together.
At each scanning position, an image of the spots irradiated in the sample is recorded on a detector, which therefore records an array of image light spots. A complete image of the sample is constructed by processing the various recorded images, the processing being performed once all the images have been recorded or in real time; in this latter case the complete image is completed as the images are recorded. Therefore, when the array of light spots is scanned with respect to the
detector, the corresponding positions of the focused light spots in the sample have to be determined, in order to build a correct complete image from the images recorded at the different positions.
The term image should be understood in a broad sense. An image comprises information on the different irradiated points of the sample. The information may contain the amount of reflected light at each point, the amount of scattered light at each point, etc. Imaging permits to measure the spatially extended structural properties of the sample. In the description, the term "image" will be used to designate:
- an image, on the detector, of a light spot, which will be called an "image light spot";
- an image, on the detector, of the array of light spots at a particular position, which is recorded in the device and which will be called a "recorded image";
- a complete image of the sample (or of part of the sample), which is constructed on the basis of the recorded images and which will be called a "complete image".
The microscope comprises means for displacing the array of light spots with respect to the sample. Each recorded image corresponds to a particular position of those displacing means and therefore to a corresponding position of the array of light spots within the sample. In order to obtain those recorded images, the displacing means are displaced step by step by predetermined distances and an image is recorded at each predetermined position, or the displacing means are displaced continuously (for instance at a constant speed) and an imaging activator records images at each predetermined position. Each image is attributed to a predetermined position in the sample. However, the displacing means and/or the imaging activator may suffer from lack of accuracy. If the array of light spots is not at the correct predetermined position when the image is taken, the information on the detector will be attributed to incorrect points in the sample, therefore leading to a distorted image.
The actual displacement of the means for displacing the array of light spots should therefore be determined. According to a solution, measuring means may be connected to the means for displacing the array of light spots, so as to measure their actual displacement; however, such a solution is not necessarily very accurate and is not easy to implement. Another solution is to have very accuracy displacing means that
can be trusted in; however, such displacing means are expensive. Another solution for determining the position of the array of light spots in the sample consists in performing a measurement of the position of an element of the microscope, which position is related to the relative movement between the array of light spots and the sample. For instance, the chosen element can be the holder of the sample, if this latter is moved during the scanning operation (the array of light spots being fixed); in such a case, the position of a particular array of light spots in the sample is deduced from the position of the holder at the time the particular array was irradiated on the sample. Again, this solution is not very accurate.
Summary of the invention
It is therefore an object of the present invention to provide a method and device for attributing, to the recorded images of an array of light spots on the detector of a multi-spot scanning optical device, the corresponding positions of the array of light spots in the sample.
In accordance with the present invention there is provided a method for imaging a sample with a multi-spot scanning microscope, comprising:
- generating an array of light spots,
- irradiating the sample with the array of light spots, the light spots interacting with the sample and being imaged on a detector, forming an array of image light spots,
- scanning the array of light spots into a plurality of positions in the sample,
- for each position of the array of light spots in the sample, recording an image of the corresponding array of image light spots on the detector,
- calculating the relative positions of the array of image light spots on the detector between the recorded images,
- calculating the relative positions of the array of light spots in the sample between the recorded images, on the basis of the calculated relative positions of the array of image light spots on the detector, and
- constructing a complete image of the sample on the basis of the images recorded on the detector, the recorded images being attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample.
Thanks to the invention, the relative positions of the array of image light spots on the detector are calculated and the corresponding relative positions of the array of light spots irradiated in the sample are deduced therefrom. In other words, rather than attributing, to a recorded image, a predetermined position of the light spots in the sample, the invention proposes to attribute, to relative positions of the light spots on the detector between the images, relative positions of the light spots in the sample. With the method of the invention, it is not the absolute position of the array of light spots that is looked for (even if this is possible according to an embodiment of the invention) but the relative position, that is to say, the position relatively to the other positions, in other words, the evolution of the positions between the images. The evolution of the positions depends on the actual positions of the means for scanning the array of light spots in the sample when the images are recorded. Therefore, by calculating the relative positions of the array of light spots in the sample, when reconstructing the complete image of the sample on the basis of the images recorded on the detector, the recorded images are attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample. In other words, the positions - to which the recorded images are attributed - are placed the ones relatively to the others, i.e. the positions are spaced apart the ones from the others, by taking into account their calculated relative positions. This permits to avoid distortions of the complete image; indeed, distortions would occur if the recorded images were attributed to wrong positions of the array of light spots in the sample relatively the ones to the others. The result of the method of the invention is that the proportions of the sample are kept in the complete image that is constructed.
By complete image of the sample, what should be understood is the complete image of a whole object or of part of an object (this part of the object therefore being considered as the sample).
According to an embodiment, the calculated relative positions of the array of light spots in the sample are the same as the corresponding calculated relative positions of the array of image light spots on the detector.
According to an embodiment, the relative positions of the array of image light spots on the detector are calculated on the basis of the relative positions of a plurality of light spots of the array that are averaged over the image light spots of the array.
According to an embodiment, the method comprises:
- calculating the position, in at least two successive images on the detector, of at least one image light spot.
According to an embodiment, the relative positions being calculated on the basis of the distances between successive positions, the method comprises:
- calculating the distance between the calculated positions. According to an embodiment, the method comprises:
- calculating the position of at least one image light spot in more than two successive images on the detector and calculating the distance between the calculated positions,
- calculating the ratio between successive calculated distances between successive images and
- multiplying respectively the distances between the positions of the array of light spots in the sample to which the recorded images are attributed when constructing the complete image of the sample, by the calculated ratios.
According to an embodiment, the method comprises:
- calculating, for each image, the position of a plurality of image light spots of the array on the detector and
- calculating a distance ratio averaged over the light spots of the array. According to an embodiment, the optical device comprising a pixelated detector, which comprises a plurality of pixels adapted to be activated by the image light spots, the position of an image light spot is calculated as the position of at least one pixel activated by said image light spot.
According to an embodiment, an image light spot activating a plurality of pixels, the position of the image light spot is calculated as the average position of the activated pixels.
According to an embodiment, the pixels being activated at a certain intensity depending on the intensity of the corresponding image light spot, the average position of the activated pixels is weighted by the respective intensity of their activation.
According to an embodiment, the distances on the detector are calibrated, that is to say that, to a distance on the detector, calculated in a number of pixels, an actual distance on the sample is attributed.
According to an embodiment, the relative positions of the array of image light spots are calculated as the average positions of the image light spots on the detector, in arbitrary units.
According to an embodiment, the array of light spots comprising spots aligned in two directions, the positions into which the array of light spots is scanned substantially form a straight line that makes an angle with the directions of the array.
According to an embodiment, the array of light spots comprising spots aligned in two directions, the positions into which the array of light spots is scanned substantially form a set of straight segments, each straight segment having a length substantially equal to the distance between two neighboring light spots in the array and making substantially no angle with one direction of the array, the adjacent straight segments being joined by intermediate segments.
In accordance with the present invention there is also provided a multi-spot scanning microscope comprising:
- means for generating an array of light spots which are intended to irradiate a sample,
- means for imaging the light spots irradiating the sample on a detector, the imaged light spots forming an array of image light spots,
- means for scanning the sample with the array of light spots according to a plurality of positions,
- means for recording, for each position of the array of light spots in the sample, an image of the corresponding array of image light spots on the detector,
- means for calculating the relative positions of the array of image light spots on the detector between the recorded images,
- means for calculating the relative positions of the array of light spots in the sample between the recorded images, on the basis of the calculated relative positions of the array of image light spots on the detector, and
- means for constructing a complete image of the sample on the basis of the images recorded on the detector, the recorded images being attributed to corresponding
positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample.
According to an embodiment, the device comprises a first pixelated detector with high magnification and a second pixelated detector with low magnification.
According to an embodiment, the device comprises means for blurring the image of the image light spots on the detector.
According to an embodiment, the device is a multi-spot scanning microscope.
In accordance with the present invention there is also provided a method for processing a set of recorded images in order to form an image of a sample, each recorded image representing an array of light spots on a detector forming the image of an array of light spots irradiated in the sample, each recorded image corresponding to a different position of the array of lights spots in the sample, the method comprising:
- loading the set of recorded images,
- calculating the relative positions of the array of image light spots on the detector between the recorded images,
- calculating the relative positions of the array of light spots in the sample between the recorded images, on the basis of the calculated relative positions of the array of image light spots on the detector, and
- constructing a complete image of the sample on the basis of the images recorded on the detector, the recorded images being attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample.
Therefore, the images on the detector may be recorded on a suitable medium and the set of recorded images may be provided to data processing means carrying out the above method.
In accordance with the present invention there is also provided a computer program adapted to carry out the above method.
In accordance with the present invention there is also provided a computer program product comprising program code for execution of the above method.
These and other aspects of the invention will be more apparent from the following description with reference to the attached drawings.
Brief description of the drawings
- Fig.1 is a schematic diagram of a multi-spot scanning microscope according to the invention;
- Fig.2 shows an image, on the detector of the microscope of Fig.1, of an array of light spots irradiated in the sample;
- Fig.3 shows a zoomed- in image of a single image light spot at a first position on the detector and
- Fig.4 shows a zoomed-in image of the single image light spot of Fig.3 at a second position on the detector.
Detailed description of embodiments
With reference to Fig.1 , an embodiment of an optical scanning microscope 1 according to the invention comprises a light source (not represented), generating a parallel light beam 2 propagating along the optical axis z of the microscope 1. The light beam 2 is incident on a diffractive structure 3, such as a binary phase structure. The diffractive structure 3 generates a multitude of orders of diffraction in numerous directions, the orders interfering and creating at a certain distance an array 4 of light spots S. The light source and the diffractive structure 3 form means for generating an array of light spots 4. Any other suitable means may be contemplated for generating an array of light spots 4.
The focused light spots S of the array 4 are irradiated on a sample 5, which is positioned on a sample support 6, herein comprising for example two glass plates (or slides), as known in the art. The irradiated light spots S form an array 4 in two perpendicular directions, for instance in x and y directions or two perpendicular directions transverse to the direction z of the optical axis and having an angle with the x and y directions. Each light spot is separated from the nearest light spots by a pitch, which is in this embodiment equal in the two directions of the array 4. Of course, other orderings of the light spots within the array 4 may be contemplated, notably more complex configurations.
The irradiated light spots interact with the sample 5 and are imaged on a pixelated detector (sensor) 8 onto which they are focused by imaging optics 9, thereby
forming on the detector 8 an array 7 of image light spots S'. The structure of the imaging optics will not be detailed as it is easily accessible to the person skilled in the art. An image light spot S' should be understood as the image, on the detector 8, of a light spot S irradiated in the sample 5.
According to an embodiment, the detector 8 is provided on the same side as the side of incidence of the array of irradiated light spots 4, with appropriate optics for irradiation and detection of the light spots on the same side of the sample 5. According to another embodiment, which is the embodiment represented in Fig.l, the detector 8 is provided on the opposite side, the light passing though the sample 5 and its support 6.
In the embodiment described, each light spot S irradiated on the sample 5 is imaged into a single image light spot S' on the detector 8. Therefore, the array 4 of irradiated light spots S is imaged into an array 7 of image light spots S', the two arrays 4, 7 having a global similar shape, an example of which can be seen on Fig.2 which shows image light spots S' on the detector 8.
Each image light spot S' is analyzed individually to measure a particular feature, for instance its intensity, in order to deduce therefrom a particular characteristic of the point of the sample 5 corresponding to the image light spot S', that is to say, the point of the sample 5 that has been irradiated by the light spot S which, interacted with the sample 5, has been imaged as the image light spot S' on the detector 8. As an example, attenuation in the intensity of an image light spot S' may reveal the presence of an absorbing substance at the corresponding point of the sample 5.
The array of irradiated light spots 4 may be scanned into a plurality of positions in the sample 5. At each position of the array 4 in the sample 5, a corresponding image is recorded on the detector 8; the recorded image comprises the array 7 of image light spots S'. An example of such a recorded image is shown on Fig.2. Each image light spot S' of this array 7 comprises information on the sample 5, as explained above.
A complete image of the sample 5 (or of a certain surface of the sample 5) is constructed on the basis of all the recorded images. Indeed, if the array of irradiated light spots 4 has been scanned into a plurality of positions within all the sample 5 (or within all said certain surface of the sample 5), an image being recorded at each position, the plurality of images contain image light spots S' representing all the
positions in the sample 5 (or all the positions in said certain surface of the sample 5). Constructing the complete image may be performed after all the images have been recorded or in real time as the images are recorded.
In order to scan the sample 5, the microscope 1 comprises means for scanning the array 4 of light spots S irradiated in the sample 5. For instance, those means may comprise means for displacing, relatively to the sample 5, the means for generating the array of irradiated light spots 4. According to the embodiment described, the microscope 1 comprises means for displacing the diffractive structure 3 relatively to the sample 5.
In order to scan the sample 5, the array of irradiated light spots 4 is scanned along two directions, for instance the two directions of the array, into a plurality of positions, as explained above; the displacing means may be displaced whether step by step or continuously. In case the dimensions of the array 4 are bigger than the dimensions of the surface of the sample 5 to be scanned, the array 4 only needs to be scanned over the surface of an elementary cell of the array 4, defined by four proximal light spots S, to scan the whole sample surface; indeed, since all the irradiated light spots S are displaced together, if each spot covers the elementary cell, then the array 4 covers the whole scanned surface of the sample 5. In case the dimensions of the array 4 are smaller than the dimensions of the sample 5, the array 4 may be scanned over the surface of the sample 5 that is in the field of view of the imaging optics 9 of the microscope 1 and an image is taken from that surface, and optionally the sample 5 is displaced so as to scan another surface of it.
In any case, the array of irradiated light spots 4 is scanned relatively to the sample 5 in such a manner that each required point of the sample 5 (depending on the required resolution) is touched at least once by a light spot S, in one of the scanning positions. The resolution is determined by the size of the light spots and the distance between two positions of the light spots, that is to say, the distance between two successive positions. For instance, the light spots can be of a diameter equal to lμm, spaced apart in the array by 15μm, the array being moved by lμm between two positions of the array 4 so as to scan the entire surface of the sample 5. The array can also be displaced in such a way that the entire surface of the sample 5 is not scanned but only a discrete portion of it is scanned, for instance if 1 μm diameter light spots are
displaced, between two successive positions, by 2μm. According to another embodiment, oversampling is performed, that is to say, the displacement steps between two successive positions of the light spots are smaller than the diameter of the light spots, for instance lμm diameter light spots are displaced, between two successive positions, by 0.5μm; oversampling permits to get a better resolution. According to an embodiment, so-called Nyquist-sampling is performed, where the sampling distance (displacement step) is equal to R/2 for a resolution equal to R, R being determined by the optical cutoff (R=λ/(2*NA) where λ is the wavelength and NA the numerical aperture of the device).
Incidentally, if the two directions of the array 4 make an angle with the translation directions of the means for scanning the array 4, a one direction translation scanning may be sufficient to scan all the positions in the sample 5, for instance if two adjacent light spots are offset, preferably of half the diameter of a light spot (such a configuration leads to an image, on the detector 8, comparable to the one of Fig.2, where it should be contemplated that the array is displaced horizontally between two positions). In other words, the array (4) of light spots (S) comprising spots (S) aligned in two directions, the positions into which the array (4) of light spots (S) is scanned substantially form a straight line that makes an angle with the directions of the array
(4).
According to another embodiment, the array (4) of light spots (S) comprising spots (S) aligned in two directions, the positions into which the array (4) of light spots (S) is scanned substantially form a set of straight segments, each straight segment having a length substantially equal to the distance between two neighboring light spots (S) in the array (4) and making substantially no angle with one direction of the array (4), the adjacent straight segments being joined by intermediate segments; those intermediate segments may be parallel to the second direction of the array.
On Fig.l, one particular irradiated light spot S is represented, in three successive positions Pl, P2, P3 of the array of light spots 4 in the sample 5 (Fig.l is not on scale, for the sake of clarity). From the first position Pl to the second position P2 of the light spot S in the sample 5, the array 4 of irradiated light spots S has been displaced by a first displacement distance dl. From the second position P2 to the third
position P3 of the light spot S in the sample 5, the array 4 has been displaced by a second displacement distance d2 in the sample 5.
The light spot S irradiated on the sample 5 is imaged on the detector 8 as an image light spot S'. Each position Pl, P2, P3 of the irradiated light spot S in the sample 5 results in a position P'l, P'2, P'3 of the image light spot S' on the detector 8. From the first position P'l to the second position P'2 on the detector 8, the array 7 of image light spots S' has been displaced by a first displacement distance d'l. From the second position P'2 to the third position P'3 on the detector 8, the array 7 has been displaced by a second displacement distance d'2 on the detector 8.
The means for displacing the diffractive structure 3 may not necessarily be very accurate. Let us consider that the nominal predetermined displacement distance between two successive positions of the array 4 should be 8μm. For instance, between the first and second positions Pl, P2, the means for displacing the array 4 may have displaced the array 4 by a distance dl=8μm, while between the second and third positions P2, P3, it may have displaced the array 4 by a distance d2=10μm, although d2 should theoretically have also been equal to the predetermined distance 8μm. The three successive images on the detector 8, corresponding to said positions Pl, P2, P3, should therefore not be attributed to points in the sample 5 spaced apart by equal distances, otherwise distortions in the constructed complete image will occur.
This is why the method of the invention comprises the following steps:
- for each position of the array of light spots in the sample, recording an image of the corresponding array of image light spots on the detector,
- calculating the relative positions of the array of image light spots on the detector between the recorded images,
- calculating the relative positions of the array of light spots in the sample between the recorded images, on the basis of the calculated relative positions of the array of image light spots, and
- constructing a complete image of the sample on the basis of the images recorded on the detector, the recorded images being attributed to corresponding positions of the array of light spots in the sample, these positions taking into account the calculated relative positions of the array of light spots in the sample.
By relative, what should be understood is something that is not absolute, that depends on something. Therefore, a relative position is a position compared to something, that is to say, relatively to something, here, relatively to the other positions.
According to an embodiment, the relative positions are calculated on the basis of the distances between two successive positions.
If the optical device is a multi-spot scanning microscope comprising a pixelated detector 8, which comprises a plurality of pixels that may be activated by the image light spots S', the relative positions of the image light spots S' are calculated as the relative positions of the pixels activated by the corresponding image light spots S'. Indeed, an image light spot S' activates at least one pixel, with a certain intensity.
In the embodiment described, the displacement distances d'l, d'2 between successive positions of the image light spots S' on the detector 8 are used to calculate the relative positions of the array 4 of light spots S in the sample 5.
According to an embodiment, a relative position is calculated as the ratio between successive distances d'l, d'2 between successive positions P'l, P'2, P'3. The relative positions of the array 7 of image light spots S' between successive images are here attributed to the array 4 of irradiated light spots S, that is to say, the relative positions of the array 4 of irradiated light spots S in the sample 5 are set equal to the ratios calculated for the array 7 of image light spots S'.
Indeed, and as already explained above, what is interesting is not the actual values of the distances dl, d2 as such (even if it is possible to calculate those values by further using calibration, as will be explained below). What is interesting is the relation between those distances dl, d2, in the embodiment described, their ratio. Obtaining the ratio between successive distances dl, d2 corresponding to successive images permits, when forming the complete image of the sample 5, to attribute the different recorded images to positions in the sample 5 that are correctly positioned relatively to each other and, thereby, to avoid distortions by keeping the proportions between the distances dl, d2 in the sample. In the complete image, the image light spots S' will be spaced apart by correct distances, the calculated ratios being respected.
Fig.3 and 4 each represent a position P'l, P'2 of a single image light spot S' on the detector 8. Each position P'l, P'2 corresponds to an image recorded on the detector 8. Those positions P'l, P'2 are successive positions, corresponding to successive
images. Let us suppose that, between the first and second positions P'l, P'2, there is a displacement distance d'l of 12 pixels. Let us also suppose that between the second and third position P'2, P'3 (not shown on Fig.3 and 4 but shown on Fig.l), there is a displacement distance d'2 equal to 15 pixels. From this distance calculation, it can be concluded that the second displacement distance d'2 is 25% bigger than the first displacement distance d'l. The ratio between the distances d'l, d'2 on the detector 8 is therefore equal to 1.25. This ratio is applied to the corresponding positions Pl, P2, P3 (of the array 4 of irradiated light spots S) that are attributed to the recorded images when constructing the complete image of the sample 5. In other words, in the step of construction of the complete image of the sample 5, the images respectively corresponding to the first, second and third positions P'l, P'2, P'3 on the detector 8 are attributed to positions Pl, P2, P3 in the sample 5 observing the calculated relative positions, that is to say, the calculated ratio of the distances between the positions P'l, P'2, P'3. Therefore, the distance d2 between the second and third positions P2, P3 is 1.25 times the distance dl between the first and second positions Pl, P2. Distortions are hence compensated for.
Imaging can therefore be performed, which measures the spatially extended structural properties of the sample 5.
In the embodiment which has been described, the relative positions are calculated on the basis of the distances between successive positions, between successive images. Other relative positioning can of course be contemplated.
According to an embodiment, the relative positions of the array 7 of image light spots S' are calculated as the averaged positions of the light spots S', in arbitrary units. In other words, the positions of all the image light spots S' are calculated and those positions are averaged in order to give a certain position, which is the average position of the spots. For instance, if each image light spot S' on the detector 8 has coordinates (x,y), the x values of all the image light spots S' are summed and averaged and the y values of all the image light spots S' are summed and averaged, those averaged values giving an average position in arbitrary units. This is done for each recorded image and, from those relative positions in arbitrary units on the detector 8, the relative positions of the array 4 of irradiated light spots S in the sample 5 can be determined. The arbitrary units can in fact be considered as relative coordinates. It
should be noticed that, in the calculation of the averaged position in arbitrary units, the image light spots S' situated in the outer bands of the array 7 may not be taken into account if there is a risk that, during the scanning, a spot disappears or appears on a side of the detector between two images; at the contrary, the positions can only be compared between the images if the same spots S', having been displaced, are taken as the basis of the determination of the relative position of the array 7.
According to a particular embodiment of the invention, the means for scanning the array 4 of irradiated light spots S may be cheap ones, since even if those means move the array 4 of light spots S by irregular distances, these irregularities will be compensated for thanks to the method of the invention.
It shall be noted that the method of the invention applies to a relative movement between two assemblies: the means for generating and displacing the array 4 of light spots S on the one hand, the sample 5 and the detector 8 on the other hand. If the sample 5 were moved relatively to the detector 8, the means for generating and displacing the array 4 of light spots S being fixed, the detector 8 would indeed measure information on different positions in the sample 5, but the image light spots S' would appear at the same positions on the detector 8 for every images, therefore forbidding to perform the method of the invention, which establishes a relation between the relative positions on the detector 8 and the relative positions in the sample 5.
As said before, in multi-spot scanning microscopy, if the detector 8 is pixelated, the relative positions of the array 7 of image light spots S' between the images can easily be calculated on the basis of the relative positions of the corresponding activated pixels.
According to an embodiment, the relative positions of the pixels activated by a unique particular image light spot S' between the images are calculated and attributed to the whole array 7 of image light spots S' and, therefore, to the whole array 4 of irradiated light spots S. In other words, the calculations are made for only one image light spot S' between successive images, then considering that the light spots S' of the array 7 of image light spots S' are all displaced by the same distances as this particular light spot S', which are therefore considered as the displacement distance of the array 7. In other words again, it implies considering that all the light spots S' are displaced as a block, without any relative displacements between them.
According to another embodiment, the relative positions of a plurality of image light spots S' (or corresponding activated pixels) between images are calculated, those relative positions being averaged. According to a particular embodiment in that case, the relative positions of all the image light spots S' are averaged.
Taking the example of the above mentioned embodiment where the ratio of the displacement distances between successive positions is calculated, the method comprises:
- calculating the position of a plurality of image light spots of the array and
- calculating a ratio averaged over the light spots of the array.
In order to obtain an averaged ratio, three embodiments can be contemplated. Firstly, the positions of the light spots S' are calculated and averaged, then the distances between successive averaged positions are calculated and their ratio is calculated. Secondly, the positions of the light spots S' are calculated, then the distances between successive positions are calculated and averaged and their ratio is calculated. Thirdly, the positions of the light spots S' are calculated, then the distances between successive positions are calculated and their ratio is calculated and then averaged.
In any case, the relative positions of the array 7 are considered as being equal to the averaged relative positions of the image light spots S' over the array 7.
An advantage of averaging the relative positions of the different image light spots S' of the array 7 is that it provides a better accuracy of the position measurement.
In order to calculate the relative positions of the array 7 of image light spots S', the actual positions of the image light spots S' on the detector 8 may be calculated. The total number of pixels N activated per image light spot S' on the detector 8 has to be at least one. According to an embodiment, a plurality of pixels are activated by each image light spot S', so that the position of the image light spots S' can be calculated with sub-pixel accuracy. According to an embodiment, the position of an image light spot S' is calculated as the average position of the pixels activated by the image light spot S', weighted by the respective intensities of the activated pixels. The more pixels per spot, the higher the accuracy of the position determination. According to an embodiment, at least four pixels are activated per image light spot S'. It can be seen on Figs.3 and 4 that each spot activates a plurality of pixels.
In such a case, the pixels activated by one image light spot S' are determined according to a predetermined criterion, such as a threshold comparison. Then, in order to take into account the weight of these pixels, an intensity is calculated for each of them, and, according to this, an average position is determined. We may notice that the averaging step applies there to the different pixels activated by a single light spot, the different calculated positions possibly being subsequently averaged over the whole array 7 of image light spots S' for calculating the relative positions of the array 7, as explained above.
Other methods for determining the position of an image light spots S' on the basis of the activated pixels can of course be contemplated such as, for instance, cross- correlation.
Different embodiments for activating more that one pixel per image light spot S' on the detector 8 are proposed.
According to a first embodiment, the multi-spot scanning microscope 1 comprises two detectors (not represented), a first pixelated detector with high magnification and a second pixelated detector with low magnification. A subset of image light spots S' are imaged onto the first detector, which ensures enough pixels per spot for accurate position determination. The complete array of image light spots S' is imaged at the second detector, which allows for a large field. In other words, the second detector is used for global position determination and the first detector with a subset of light spots with many pixels is used for gathering intensity information used for imaging, as well as for determining precise positions of the image light spots. To implement this first embodiment, a beam splitter can be provided that separates the light beams into two paths, one on each detector.
According to a second embodiment, the multi-spot scanning microscope 1 comprises means for blurring the image light spots S' on the detector 8. Those means may for instance comprise defocus means or means for creating spherical aberration, as well known in the art. As a consequence, the image light spot size is enlarged on the detector 8, which activates more pixels and permits to determine the position of the spots more accurately, without lowering the resolution of the microscope. Indeed, since the resolution of an image produced with the multi-spot scanning microscope 1 is determined by the quality of the light spots S irradiated in the sample 5, the blurring in
the imaging system does not lower the overall resolution of the system. The maximum blur that is allowed is when two adjacent image light spots S' imaged on the detector 8 can still be separated. Besides, the blurring does not cause any problem for gathering information from the image light spots, since even if the spot intensity is spread over a larger number of pixels, its value is the same; therefore, by integrating the intensity over all the pixels, the value of the intensity is recovered.
According to an embodiment, the distances on the detector 8 are calibrated, that is to say that an actual distance on the sample 5 is attributed to a distance on the detector 8 (calculated in a number of pixels).
Thanks to this calibration, the numerical positions and/or distances on the detector 8 can be calculated in order to give a scale to the constructed image of the sample 5. In such a case, from the positions of all spots S' in an image, an average can be taken to give the position of the array 4 of light spots S in the sample 5 with very high accuracy (sub lOOnm). Therefore, not only the image is not distorted, but also its scale is known.
The method of the invention in particular permits real-time calibration.
Another way of attributing a distance on the detector to a distance in the sample is to know the magnification of the imaging optics.
Another simple way to calibrate the distances on the detector 8 is to use the known distance between two successive light spots of the array 4 of light spots S irradiated on the sample 5. If this distance is known - which is usually the case since the means for generating the array 4 of light spots S are calibrated - therefore the calibration is easily obtained by calculating the number of pixels separating two successive image light spots S' and attributing, to this number of pixels, the known distance.
According to an aspect of the invention, once the array (4) of light spots (S) has been imaged on the detector (8), the images may be recorded on a suitable medium and the set of recorded images may be provided to data processing means, adapted to process the recorded images in order to form a complete image of the sample (5).
A computer program may be provided for enabling the data processing means to carry out the processing method. The computer program may be recorded on a computer program product.
While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.
Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid- state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.
Claims
Claims
1- Method for imaging a sample (5) with a multi-spot scanning microscope, comprising:
- generating an array (4) of light spots (S),
- irradiating the sample (5) with the array (4) of light spots (S), the light spots (S) interacting with the sample (5) and being imaged on a detector (8), forming an array of image light spots (S'),
- scanning the array (4) of light spots (S) into a plurality of positions (Pl, P2, P3) in the sample,
- for each position (Pl, P2, P3) of the array (4) of light spots (S) in the sample (5), recording an image of the corresponding array (7) of image light spots (S') on the detector (8),
- calculating the relative positions of the array (7) of image light spots (S') on the detector (8) between the recorded images,
- calculating the relative positions of the array (4) of light spots (S) in the sample (5) between the recorded images, on the basis of the calculated relative positions of the array (7) of image light spots (S') on the detector (8), and
- constructing a complete image of the sample (5) on the basis of the images recorded on the detector (8), the recorded images being attributed to corresponding positions of the array (4) of light spots (S) in the sample (5), these positions taking into account the calculated relative positions of the array (7) of light spots (S) in the sample (5).
2- Method according to claim 1, wherein the calculated relative positions of the array (4) of light spots (S) in the sample (5) are the same as the corresponding calculated relative positions of the array (7) of image light spots (S') on the detector (8).
3- Method according to claim 1, wherein the relative positions of the array (7) of image light spots (S') on the detector (8) are calculated on the basis of the relative positions of a plurality of light spots (S') of the array (7) that are averaged over the image light spots (S') of the array (7).
4- Method according to claim 1, comprising:
- calculating the position (P' 1, P'2, P'3), in at least two successive images on the detector (8), of at least one image light spot (S').
5- Method according to claim 4 wherein, the relative positions being calculated on the basis of the distances ((dl, d2), (d'l, d'2)) between successive positions ((Pl, P2, P3), (P' 1, P'2, P'3)), the method comprises:
- calculating the distance (d'l, d'2) between the calculated positions (P' 1, P'2, P'3).
6- Method according to claim 4 comprising:
- calculating the position (P' 1, P'2, P'3) of at least one image light spot (S') in more than two successive images on the detector (8) and calculating the distance (d'l, d'2) between the calculated positions (P' 1, P'2, P'3),
- calculating the ratio between successive calculated distances (d'l, d'2) between successive images and
- multiplying respectively the distances (dl, d2) between the positions (Pl, P2, P3) of the array (4) of light spots (S) in the sample (5) to which the recorded images are attributed when constructing the complete image of the sample (5), by the calculated ratios.
7- Method according to claim 6 comprising:
- calculating, for each image, the position of a plurality of image light spots (S') of the array (7) on the detector (8) and
- calculating a distance ratio averaged over the light spots (S') of the array (7).
8- Method according to claim 4 wherein, the optical device comprising a pixelated detector (8) which comprises a plurality of pixels adapted to be activated by the image light spots (S'), the position of an image light spot (S') is calculated as the position of at least one pixel activated by said image light spot (S').
9- Method according to claim 8 wherein, an image light spot (S') activating a plurality of pixels, the position (P' 1, P'2, P'3) of the image light spot (S') is calculated as the average position of the activated pixels.
10- Method according to claim 9 wherein, the pixels being activated at a certain intensity depending on the intensity of the corresponding image light spot (S'), the average position of the activated pixels is weighted by the respective intensity of their activation.
11- Method according to claim 4, wherein the relative positions of the array (7) of image light spots (S') are calculated as the average positions of the image light spots (S') on the detector (8), in arbitrary units.
12- Multi-spot scanning microscope comprising:
- means (2, 3) for generating an array (4) of light spots (S) which are intended to irradiate a sample (5)
- means (9) for imaging the light spots irradiating the sample (5) on a detector (8), the imaged light spots (S') forming an array (7) of image light spots (S'),
- means for scanning the sample with the array (4) of light spots (S) according to a plurality of positions (Pl, P2, P3),
- means for recording, for each position (Pl, P2, P3) of the array (4) of light spots (S) in the sample (5), an image of the corresponding array (7) of image light spots (S') on the detector (8),
- means for calculating the relative positions of the array (7) of image light spots (S') on the detector (8) between the recorded images,
- means for calculating the relative positions of the array (4) of light spots (S) in the sample (5) between the recorded images, on the basis of the calculated relative positions of the array (7) of image light spots (S') on the detector (8), and
- means for constructing a complete image of the sample (5) on the basis of the images recorded on the detector (8), the recorded images being attributed to corresponding positions of the array (4) of light spots (S) in the sample (5), these positions taking into account the calculated relative positions of the array (4) of light spots (S) in the sample (5).
13- Device according to claim 12, comprising a first pixelated detector with high magnification and a second pixelated detector with low magnification.
14- Device according to claim 12, comprising means for blurring the image of the image light spots (S') on the detector (8).
15- Method for processing a set of recorded images in order to form an image of a sample (5), each recorded image representing an array (7) of light spots (S') on a detector (8) forming the image of an array (4) of light spots (S) irradiated in the sample (5), each recorded image corresponding to a different position (Pl, P2, P3) of the array (4) of lights spots (S) in the sample (5), the method comprising:
- loading the set of recorded images,
- calculating the relative positions of the array (7) of image light spots (S') on the detector (8) between the recorded images,
- calculating the relative positions of the array (4) of light spots (S) in the sample (5) between the recorded images, on the basis of the calculated relative positions of the array (7) of image light spots (S') on the detector (8), and
- constructing a complete image of the sample (5) on the basis of the images recorded on the detector (8), the recorded images being attributed to corresponding positions of the array (4) of light spots (S) in the sample (5), these positions taking into account the calculated relative positions of the array (7) of light spots (S) in the sample (5).
16- Computer program adapted to carry out the method of claim 15.
17- Computer program product comprising program code for execution of the method of claim 15.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP07301112 | 2007-06-15 | ||
EP07301112.4 | 2007-06-15 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2008152605A1 true WO2008152605A1 (en) | 2008-12-18 |
Family
ID=39713724
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2008/052338 WO2008152605A1 (en) | 2007-06-15 | 2008-06-13 | Multi-spot scanning optical device for imaging of a sample |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2008152605A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2013049646A1 (en) | 2011-09-29 | 2013-04-04 | Fei Company | Microscope device |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4769698A (en) * | 1985-10-04 | 1988-09-06 | National Biomedical Research Foundation | Interactive microscopic image display system and method |
US4806004A (en) * | 1987-07-10 | 1989-02-21 | California Institute Of Technology | Scanning microscopy |
US5239178A (en) * | 1990-11-10 | 1993-08-24 | Carl Zeiss | Optical device with an illuminating grid and detector grid arranged confocally to an object |
US20020090127A1 (en) * | 2001-01-11 | 2002-07-11 | Interscope Technologies, Inc. | System for creating microscopic digital montage images |
US6697164B1 (en) * | 1998-08-05 | 2004-02-24 | Cadent Ltd. | Imaging a three-dimensional structure by confocal focussing an array of light beams |
EP1510851A1 (en) * | 2003-08-25 | 2005-03-02 | Cadent Limited | Apparatus and method for providing high intensity non-coherent light and for speckle reduction |
US20060072191A1 (en) * | 2004-10-06 | 2006-04-06 | Yokogawa Electric Corporation | Confocal microscope |
-
2008
- 2008-06-13 WO PCT/IB2008/052338 patent/WO2008152605A1/en active Application Filing
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4769698A (en) * | 1985-10-04 | 1988-09-06 | National Biomedical Research Foundation | Interactive microscopic image display system and method |
US4806004A (en) * | 1987-07-10 | 1989-02-21 | California Institute Of Technology | Scanning microscopy |
US5239178A (en) * | 1990-11-10 | 1993-08-24 | Carl Zeiss | Optical device with an illuminating grid and detector grid arranged confocally to an object |
US6697164B1 (en) * | 1998-08-05 | 2004-02-24 | Cadent Ltd. | Imaging a three-dimensional structure by confocal focussing an array of light beams |
US20020090127A1 (en) * | 2001-01-11 | 2002-07-11 | Interscope Technologies, Inc. | System for creating microscopic digital montage images |
EP1510851A1 (en) * | 2003-08-25 | 2005-03-02 | Cadent Limited | Apparatus and method for providing high intensity non-coherent light and for speckle reduction |
US20060072191A1 (en) * | 2004-10-06 | 2006-04-06 | Yokogawa Electric Corporation | Confocal microscope |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2013049646A1 (en) | 2011-09-29 | 2013-04-04 | Fei Company | Microscope device |
CN103890633A (en) * | 2011-09-29 | 2014-06-25 | Fei公司 | Microscope device |
EP2761356A1 (en) * | 2011-09-29 | 2014-08-06 | FEI Company | Microscope device |
JP2014532197A (en) * | 2011-09-29 | 2014-12-04 | エフ・イ−・アイ・カンパニー | Microscope device |
EP2761356A4 (en) * | 2011-09-29 | 2015-02-25 | Fei Co | Microscope device |
US9606345B2 (en) | 2011-09-29 | 2017-03-28 | Fei Company | Microscope device |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6934856B2 (en) | Optical sheet microscope that simultaneously images multiple target surfaces | |
US8731272B2 (en) | Computational adaptive optics for interferometric synthetic aperture microscopy and other interferometric imaging | |
JP4943946B2 (en) | Eccentricity measuring device | |
JP4747243B2 (en) | Method and apparatus for optical grasping of samples by optical deep decomposition | |
EP2452221A1 (en) | Microscopy with adaptive optics | |
US20110025837A1 (en) | method of imaging a sample | |
US11243390B2 (en) | Apparatus and method of providing parameter estimation | |
KR20160016962A (en) | Adjustment method for adaptive optics system, adaptive optics system, and storage medium storing program for adaptive optics system | |
JP2011508255A (en) | Scanning microscope and sample imaging method | |
JP2011530708A (en) | Measurement and correction of lens distortion in a multi-spot scanner | |
JP2004219987A (en) | Multilayer observation type optical microscope and multilayer observation unit | |
Luo et al. | Area scanning method for 3D surface profilometry based on an adaptive confocal microscope | |
CN105758381A (en) | Method for detecting inclination of camera die set based on frequency spectrum analysis | |
US20230393379A1 (en) | Apparatus and method of estimating values from images | |
JPH1068616A (en) | Shape measuring equipment | |
WO2008152605A1 (en) | Multi-spot scanning optical device for imaging of a sample | |
JP2016515718A (en) | Optical microscopy for the localization of point objects | |
CN114076670B (en) | Splicing primary mirror common-phase error detection method, system and storage medium | |
US11119299B2 (en) | Area scanning confocal microscopy (ASCM) | |
DE10049296B4 (en) | Optical arrangement and method for deflecting a light beam | |
US20230069794A1 (en) | Dual-mode restoration microscopy | |
CN114371549A (en) | Quantitative phase imaging method and system based on multi-focus multiplexing lens |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 08763325 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 08763325 Country of ref document: EP Kind code of ref document: A1 |