WO2017178141A1

WO2017178141A1 - Image element

Info

Publication number: WO2017178141A1
Application number: PCT/EP2017/054092
Authority: WO
Inventors: Thomas Ußmüller; Lan SHI; Christopher Söll; Andreas Bänisch; Jürgen SEILER
Original assignee: Friedrich-Alexander-Universität Erlangen-Nürnberg
Priority date: 2016-04-14
Filing date: 2017-02-22
Publication date: 2017-10-19
Also published as: DE102016206330B3

Abstract

The invention relates to an image element (1) for determining brightness information and colour information. The image element (1) comprises, on a light-sensitive detector surface (2) divided with subsurfaces (3, 4, 5) into at least three separately readable individual detectors (8, 9, 10): a colour filter element (14) that covers the detector surface (2) and comprises colour filters (17, 18, 19) which differ in their filter properties and are associated with each of the subsurfaces (3, 4, 5); and a common electronic read-out unit (6) connected to the individual detectors (8, 9, 10) for the purpose of reading out each of said individual detectors (8, 9, 10), the subsurfaces (3, 4, 5) being connected one within another. The invention also relates to an image sensor (22) comprising a plurality of such image elements (1).

Description

description

picture element

The invention relates to a picture element for detecting both the brightness and the color of incident light. The invention further relates to an image sensor having a multiplicity of such image elements arranged side by side in a planar manner. The invention relates to the technical field of digital image processing.

An image sensor for recording digital images is a device which converts optical information, for example from an object scene optically imaged onto the sensor, into electrical information. As a rule, such an image sensor has a large number of image elements arranged side by side in a planar manner and thus forms a so-called sensor array. The picture element here is the smallest unit of the image sensor and provides a sample of the image information when reading out. In general parlance, the term "pixel" has also been used to designate a pixel, on the other hand a pixel in the digital image represents a single pixel and thus the smallest element of a digital raster graphic Pixels in the sense of a pixel of the image sensor match.

In order to represent a digital color image, basically two types of information per pixel (here: pixel) are necessary, namely a brightness value and a color value. The rasterized digital image is obtained from the brightness and color values obtained for each pixel by scanning the optical image using the pixels. Image sensors for imaging in the visible spectral range known today are semiconductor-based and usually use CCD or CMOS detectors in the picture elements. Such detectors as such provide only brightness values but no color information. Nevertheless, to capture a colored image with such detectors, various techniques are used.

The so-called Bayer filter sensor, as described for example in US Pat. No. 3,971,065 A, is constructed like a chessboard. Each picture element of the sensor array contains e.g. a photodiode that receives the brightness information. In order to capture color information, there is exactly one color filter element above each picture element, which passes the received brightness information of this color channel and filters out other colors. The complete sensor array is covered with three color filters (red, green and blue), whereby usually 50% of the color filters are green and 25% each of the color filters are red and blue. Each picture element only takes on the value of a color channel, ie either red, green or blue (RGB). The color information per pixel is incomplete, so that a post-processing of the image information obtained by means of the sensor array is required. For this purpose, the information of adjacent pixels in the sensor array is used for each pixel in the digital image. In other words, the information of a plurality of pixels of the sensor array is used for a color pixel in the digital image. In addition to RGB filters, which thus include red, green and blue color filters, other variants of color filter elements are known, for. A CYGM filter comprising cyane, yellow, green and magenta color filters, or the RGBW filter comprising red, green, blue color filters and an area without a color filter for obtaining pure brightness information.

A digital camera based on Bayer filter sensors typically contains only a single image sensor (sensor array). Microelectronics it is possible, in particular in the case of CMOS technology, in addition to the actual image or sensor element at the same time to integrate analog-to-digital converter and a digital signal processing on a sensor chip. The CMOS process and the integration of signal processing enable the production of cost-effective products. Due to the simple feasibility, the Bayer filter technology is widely used and is used today in most color image sensors. Disadvantageously, the resolution of such an image sensor is limited by the necessary to obtain the color information interpolation. The real resolution of the digital image is dependent on the color channels only 25% to 50% of the number of pixels of the sensor array related information.

In another technique, as disclosed for example in US 2009/0079834 A1, three independent individual image sensors and a dichroic prism are used. This is also referred to as a so-called 3CCD sensor. The dichroic prism divides the input light from the imaged object scene into three separate beams of different colors, such as red, green and blue. Each of the three image sensors then receives the information of a single color at full resolution. Thus, the complete color information is outputted directly each time. Post-processing of the total information output by the image sensor is not necessary in contrast to a Bayer filter sensor. To obtain the digital image, the full sensor resolution is used. Post-processing is easier. Disadvantageously, three single full-scale image sensors and a prism are needed. In addition, a complex calibration is necessary. The costs for a 3CCD sensor are much higher in comparison to a Bayer filter sensor.

The sensor of another technique for obtaining the color information, as known for example from US Pat. No. 5,965,875 A, is based on three-layer picture elements. Such a sensor is also known as the Foveon X3 sensor and is offered by the company Foveon, now Sigma. Here, the property of silicon is used that light of different wavelengths from the visible spectral range penetrates at different depths or is absorbed differently. About this depth discrimination, the color information is obtained in each pixel. Such an image sensor is also designed as a one-chip solution and usually has three output channels, namely one output channel for the colors red, green and blue. Disadvantageously, such an image sensor exhibits a comparatively high noise sensitivity in the case of ringer brightness. In addition, the noise and crosstalk of the individual color channels are different.

US 201 1/0242351 A1 discloses an image sensor which has a multiplicity of image elements (pixels) arranged next to one another in a matrix on a photosensitive detector surface. The quadrants of each picture element form subareas which are covered by various color filters and to which separate readable individual detectors are assigned. The image sensor further comprises a common readout electronics for reading the individual detectors. Further image sensors are known from US 5,148,268 A, DE 10 2012 1 10 094 A1 and US 2010/0309351 A1.

From EP 1 607 771 B1 a night vision system is known, which has a diaphragm with an optical filter. The optical filter is designed such that the diameter of the entrance pupil of the diaphragm is wavelength-dependent. The diaphragm has concentrically a translucent inner region, an almost impermeable to visible light central region with a filter layer and a translucent outer region, so that visible light is limited by a small aperture hole and near infrared light through a larger aperture hole.

The object of the invention is to provide a picture element with which, at comparatively low production costs, it is possible to detect both information about the brightness and information about the color of the incident light. Next, a corresponding image sensor is to be specified with such picture elements. In particular, the resolution of the digital color image should correspond to the resolution specified by the image sensor.

The object is achieved according to the invention by a picture element for detecting both brightness information and color information, which is a photosensitive detector surface, which is divided with sub-areas on at least three separately readable individual detectors, the detector surface covering color filter element associated with the sub-areas each, in the Filter characteristics distinguishing color filter comprises, and a common, with the single detectors interconnected read-out electronics for a respective readout of the individual detectors.

The stated object is further achieved according to the invention by an image sensor comprising a plurality of areally arranged side by side picture elements of the type described above.

Accordingly, the invention provides for each pixel to be assigned at least three individual detectors, each of which has a partial area of the total photosensitive detector area. Each of these photosensitive subareas is covered with a specifically associated color filter which passes the desired brightness information of a color channel defined by the color filter and filters out other colors. Thus, each individual detector of the picture element respectively captures the brightness information of the color channel assigned to it. Thus, the complete information about the color and brightness of the imaged image section is acquired via each picture element. As color channels, primary colors are preferably selected, e.g. Red, green and blue.

An image sensor with such picture elements has many advantages. For reading out the complete color information, that is to say the brightness values in the respective color channels, only a single sensor array is required. To reconstruct the color information, no information from neighboring pixels is needed. Thus, a color reconstruction process is eliminated. The resolution of the digital image corresponds to the real sensor resolution. In addition, the color noise is reduced due to multiple color channels per pixel. The image sensor or the entire sensor array can be implemented simply and inexpensively by means of CMOS technology. The photosensitive detector surfaces are in particular planar. Thus, the production is relatively simple and therefore cost.

In order to obtain as uniform and comparable color information in the different color channels as possible over the partial section of the optical image captured by the respective picture element, the partial areas of the individual detectors are nested according to the invention. In this way, for all color channels, a scan of the image detail captured by the picture element is uniformized over the detector surface.

In a further preferred embodiment, the nested subareas of the individual detectors are concentric to one another and insofar have a common center of gravity with regard to their geometry. Increasing the symmetry further improves the uniform sampling of the color values for all color channels. In this case, in particular, the centroids of the partial surfaces advantageously fall in each case into the center of the pixel. Thus, the image section captured by the picture element is scanned symmetrically with respect to the color values for each color channel. Our own investigations prove that the characteristics of the picture element can be further improved with regard to the scanning of the color values if the concentric partial surfaces comprise a circle and rings surrounding the circle.

The above-described concrete embodiment variants of the partial surfaces are optimized in particular in such a way that an actual mean value of the corresponding brightness is detected for each color channel from the image detail captured by the image element. An increase in the symmetry of the partial surfaces or a nested arrangement of the partial surfaces offers the greatest advantages here.

Each subarea of a single detector reads out the color information or the brightness value of a color channel. As such, the picture element provides the information for all of the selected color channels. For this purpose, each subarea of a single detector is covered with a respective associated color filter. The color filters differ in their filter properties and, in particular for different wavelengths, have a different degree of transmission, reflection and / or absorption. The color filters assigned to the at least three individual surfaces are preferably given in the sense of an RGB filter as a red, green and blue color filter. However, other color combinations may be chosen. In particular, with four individual surfaces, the above-mentioned CYGM or RGBW filters can also be formed. Each color filter covers the assigned individual area. The shape of the color filter does not necessarily correspond to the shape of the individual surface in order to ensure the covering property. In a preferred embodiment, however, corresponds to the geometric shape of a color filter, optionally except for a deviation in size, each of the geometric shape of the associated sub-area. During production, care must be taken to align the color filter element or the color filters in such a way that the respective color filter covers the partial area assigned to it.

The photosensitive detector surface or the partial surfaces of the individual detectors are expediently designed as semiconductor components. Nevertheless, the invention also covers other electronic components which exhibit a photosensitive behavior. For example, these are photoresistors, tube elements or in general components that use an external photo effect, such as photomultipliers. Again preferred are selected as individual detectors photodetectors, which exploit as such the inner and / or the outer photo effect. Light information is converted directly into electrical information. Preferred photodetectors are photomultipliers, photodiodes, CMOS detectors and CCD detectors.

The readout electronics are connected to the individual detectors or the associated subareas and, in particular, designed and set up for parallel or serial readout of the individual detectors. In the case of the possibility of a serial readout, the read-out electronics comprise, for example, a controllable switching element which switches the individual detectors to a common read-out circuit or decouples them from this. In the case of a parallel evaluation, each individual detector is assigned an evaluation circuit. Independently of this, the read-out electronics preferably have at least three output channels for reading out the brightness values of each of the at least three color channels. In an expedient development of the invention, the individual detectors are arranged in a known microelectronic production technology together on a substrate, in particular a semiconductor substrate (eg silicon), and fabricated thereon. In a preferred embodiment, the read-out electronics and the individual detectors are arranged together on the substrate and, in particular, manufactured together.

In a further preferred embodiment of the invention, the green color filter in the case of an RGB color filter element is associated with an outer subarea. In particular, this makes it possible to form the outer, green sub-area with a larger area, which reference is made to the color sensitivity of the human eye.

In a further preferred embodiment, the picture element comprises a fourth partial area which is free of a covering filter. The corresponding color filter element is transparent at this point, especially in the visible spectral range, or has no color filter there. This embodiment allows simultaneous detection of only the brightness of the incident light.

The invention further comprises an image sensor having a multiplicity of image elements of the above-described type arranged in a planar manner next to one another. The advantages mentioned for the image element and its further developments can be transferred analogously to the image sensor.

Preferably, at least two picture elements are present in the image sensor, which comprise mutually different color filter elements. The differing color filter elements are advantageously distributed irregularly or ordered over the sensor surface. This embodiment variant allows an increase in the recorded information content. Expediently, the color filter elements differ in the choice of the individual color filters. In the case of color filter elements differing on the image sensor, a downstream reconstruction method using the different sensor areas allows higher resolution to be achieved. Embodiments of the invention are explained in more detail with reference to a drawing. Showing:

Fig. 1 shows schematically a picture element with three concentric faces of

Individual detectors,

2 shows a sensor array with a multiplicity of areally arranged picture elements according to FIG. 1, and

3 shows a concrete exemplary embodiment of a picture element corresponding to FIG. 1.

FIG. 1 shows a picture element 1 for detecting both brightness information and color information. The picture element 1 comprises a photosensitive detector surface 2 which is divided into a first partial surface 3, a second partial surface 4 and a third partial surface 5. Each subarea 3, 4, 5 forms a separately readable single detector 8, 9 and 10. The individual detectors 8, 9, 10 are for example made in CMOS technology and in particular as photodiodes with a respective associated readout circuit on a substrate 12, for example made of silicon , produced. The readout circuits of the individual detectors 8, 9, 10 are combined in a readout electronics 6. The readout electronics 6 is in this case formed to a parallel or to a serial readout of the individual detectors 8, 9, 10.

The first partial surface 3 of the first single detector 8 is formed as a circle whose center of gravity coincides with the center of the pixel 1. The second partial surface 4 and the third partial surface 5 are each formed as concentric rings to the circle. Thus, the centers of gravity of the second partial surface 4 and the third partial surface 5 coincide with the center of gravity of the first partial surface 3 and thus with the center of the pixel 1. The distances between the individual partial surfaces 3, 4, 5 are production-related. The individual detectors 8, 9, 10 are separately readable. The substrate 12 and the partial surfaces 3, 4, 5 of the detector surface 2 arranged thereon are covered by a color filter element 14. The color filter element 14 has a first color filter 17, a second color filter 18 and a third color filter 19 which correspond in shape to the first partial area 3, the second partial area 4 and the third partial area 5, respectively. The first color filter 17 is a blue color filter. The second color filter 18 is a red color filter. The third color filter 19 is a green color filter. Each of the color filters 17, 18, 19 thus emits light of a respective primary color channel while blocking or absorbing light of the other color channels, respectively. The color filter element 14 is formed as an RGB filter. The choice of color filters can also be different. In addition, the selection of color filters need not be the same for each pixel.

An image section of an optical image acquired by the picture element 1 is scanned by means of the partial surfaces 3, 4, 5 with respect to its color values in a red, blue and green color channel. The brightness values of the corresponding color channels for reading out are available via corresponding connection surfaces 20 of the read-out electronics 6 on the rear side of the picture element 1. The picture element 1 is able to output the complete color information with the associated brightness values for the captured image section. In a preferred variant, switches are integrated in the read-out electronics 6 of the picture element 1, via the actuation of which the corresponding picture element 1 can be addressed for readout.

FIG. 2 shows an image sensor 22 which comprises a multiplicity of the picture elements 1 shown in FIG. 1 in a sensor array 24. The picture elements 1 are in this case arranged side by side flat in the geometry shown. A digital image scanned by the image sensor 22 has a resolution which corresponds to the real resolution with respect to the number of picture elements 1. Each picture element 1 provides complete information regarding the color values, that is to say the brightness values assigned to a color channel. Interpolation over several adjacent picture elements is not necessary to obtain color information. FIG. 3 shows a concrete exemplary embodiment of a picture element 1 according to FIG. In this case, only the individual color filters 17, 1 8, 1 9 of the color filter element 14 are shown for simplicity. The respective associated sub-areas 3, 4, 5 have the same geometry, as shown in FIG.

The first partial surface 3 lies in the center C of the picture element 1. It is designed as a circle with radius R _s and a surface S _s , and covered by the blue color filter 1 7. The second subarea 4 of the second individual detector 9 is arranged around the first subarea 3 as a ring with center C and a width W _m . The second partial surface 4 is covered by a red color filter 18. The third subarea 5 of the third single detector 10 is also referred to as a ring with center C and a width W | arranged around the second partial surface 4. The third sub-area 5 is covered by a green color filter 1 9. By way of example:

2R _s > ε, R _s : radius of 1. Subarea;

G> S, G: distance between every two faces;

R ^: Inner radius of the second partial surface;

WJR ^: width / outer radius of the second part;

^R \ i = ^R n » ^{+ G} R _u : inner radius of the third partial surface;

WR _lo : width / outer radius of the third partial surface;

W _p = R _lo + G / 2; p: Half-length of the picture-theme;

S _{ / S _m / S _s : area of 3/2/1. Subarea;

In particular, the ratio of the partial areas may also be different. Here, ε is the minimum width of a partial area and δ is the minimum distance between the partial areas depending on the semiconductor process.

The above-described picture element according to FIGS. 1 and 3 as well as a sensor array with an architecture according to FIG. 2 were verified with a test image and compared with a Bayer filter sensor of the same number of picture elements. It was a digital image with a resolution of 4.935 * 2.91 0 Pixels simulated as a real scene. In comparison with the simulated real scene, the resolution of the sensor array was much smaller, namely with 154 ^* 90 pixels selected. Therefore, each pixel can capture an image area of 32 ^* 32 original pixels. For the Bayer filter sensor, an average of the brightness of the entire area of a particular color channel is simulated as a pixel value. Thereafter, two interpolation methods were used to reconstruct the color image in the simulation. For the above-described novel picture element, three average values from the three circular or annular partial areas from the respective color channel are simulated. Therefore, an RGB color image can be directly output. The simulation result of a Bayer filter sensor shows bilinear interpolation a visibly blurred image due to the interpolation process, whereby the color noise at each edge is clearly pronounced. The simulated image of a Bayer filter sensor using gradient-corrected linear interpolation, which is significantly more complex than the bilinear interpolation, shows improved image quality. Nevertheless, overdrawings of edges occur. The color noise is also strong. For the image from a simulation with the above-described picture element, as mentioned, no reconstruction process is necessary. The three color channels (red, green and blue) are read out directly. The simulated output image remains sharp, rich in detail and high resolution. The color noise is significantly lower than in the simulated images according to the Bayer filter sensor.

All three results were compared using two objective image evaluation methods. The results are significantly improved for the novel image element or an image sensor constructed therefrom in comparison to the results with a Bayer filter sensor. While a Bayer filter with bilinear interpolation has a PSNR of 19.37 dB and a FSI Mc of 0.75 feature similarity index chrominance and a PSNR of 20.09 dB with gradient-corrected linear interpolation FSIM _C of 0.80, the image with the new picture element or image sensor has a PSNR of 21.95 dB and a FSIM _C of 0.84. The PSNR value measures the signal-to-noise ratio. A higher value implies a better signal or image quality. The FSIMc value is the result of a mathematical image quality assessment. division with respect to a reference color image adapted to the human visual system. A description of this can be found, for example, in L.

Zhang, L. Zhang, X. Mou and Z. Zhang, "FSIM: A Feature Similarity Index for Image Quality Assessment" in IEEE Transactions on Image Processing, Vol. 20, No. 8, pp. 2378-2386, Aug. 201 1 (doi: 10.1 109 / TIP.201 1 .2109730).

LIST OF REFERENCE NUMBERS

1 picture element

2 detector surface

3 first part surface

4 second part surface

5 third part surface

6 readout electronics

8 first single detector

9 second single detector

10 third single detector

12 substrate

14 color filter element

17 first color filter

18 second color filter

19 third color filter

20 connection pad

22 image sensor

24 array

Claims

claims

1 . Image element (1) for detecting brightness information and color information, comprising a photosensitive detector surface (2) which is divided by partial surfaces (3, 4, 5) onto at least three separate readable individual detectors (8, 9, 10), with a detector surface (2) covering the color filter element (14), which comprises the color surfaces (17, 18, 19) assigned to the partial surfaces (3, 4, 5) and differing in the filter properties, and having a common, with the individual detectors (8, 9, 10) interconnected readout electronics (6) to a respective readout of the individual detectors (8, 9, 10), wherein the partial surfaces (3, 4, 5) are nested.

2. picture element (1) according to claim 1, wherein the nested partial surfaces (3, 4, 5) are concentric with each other, in particular the centers of gravity of the partial surfaces (3, 4, 5) each fall into the center of the pixel (1).

3. picture element (1) according to claim 2, wherein the concentric partial surfaces (3, 4, 5) are given as a circle and as at least two rings encircling the circle.

4. picture element (1) according to any one of the preceding claims, wherein the geometric shape of a color filter (17, 18, 19) respectively corresponds to the geometric shape of the associated part surface (3, 4, 5).

5. Image element (1) according to one of the preceding claims, wherein the individual detectors (8, 9, 10) are photodetectors, in particular selected from the group comprising photomultipliers, photodiodes, CMOS detectors and CCD detectors.

6. picture element (1) according to any one of the preceding claims, wherein the readout electronics (6) to a parallel or to a serial readout of the individual detectors (8, 9, 10) is formed and arranged.

7. picture element (1) according to any one of the preceding claims, wherein the read-out electronics (6) and the individual detectors (8, 9, 10) on a common substrate (12) are arranged.

8. picture element (1) according to any one of the preceding claims, wherein the color filter element (14) comprises a red color filter (18), a green color filter (19) and a blue color filter (17).

9. picture element (1) according to claim 8, wherein the green color filter (19) is associated with an outer partial surface (5).

10. picture element (1) according to any one of the preceding claims, wherein a fourth partial surface is covered, which is free of a covering filter (17, 18, 19).

1 1. Image sensor (22) having a multiplicity of image elements (1) arranged side by side in a planar manner according to one of the preceding claims.

12. image sensor (22) according to claim 1 1, wherein at least two picture elements (1) are mutually different color filter elements (14).

The image sensor (22) of claim 12, wherein the differing color filter elements (14) differ in the choice of the individual color filters (17, 18, 19).