WO2016066308A1

WO2016066308A1 - Method and device for correcting incorrect pixel data

Info

Publication number: WO2016066308A1
Application number: PCT/EP2015/070136
Authority: WO
Inventors: Michael Schmidt
Original assignee: Basler Ag
Priority date: 2014-10-27
Filing date: 2015-09-03
Publication date: 2016-05-06
Also published as: DE102014015886B4; DE102014015886A1

Abstract

The invention relates to a device and to a method for correcting pixel data in an original image signal (B_O) obtained from an image sensor having one or two dimensional pixel data arrangements, said original image sensor (B_O) being filtered using a low pass characteristic in order to generate a reference image signal (B_R). The reference image signal (B_R) is compared to the original image signal (B_O) in order to determine a fluctuation between the reference image signal (B_R) and the original image signal (B_O). Subsequently, the pixel data of the original image signal (B_O) is then mixed with corresponding pixel data of the reference image signal (B_R) when the fluctuation between the pixel data of the original image signal (B_O) and corresponding pixel data of the reference image signal (B_R) is in a predetermined range about a predetermined threshold value.

Description

Method and device for correcting a faulty

Pixel Information The invention relates to an apparatus and a method for correcting defective pixel information from an image sensor having a one-dimensional or two-dimensional pixel arrangement.

For electronic capture of images nowadays mostly digital cameras with image sensors are used. With the aid of such digital cameras, brightness images are taken in which brightness values are assigned to the individual pixels (hereinafter referred to as pixels). The image sensor usually consists of a one-dimensional or two-dimensional periodic arrangement of pixels. Irrespective of the technology, the pixels have in common that they each have a photosensitive region, for example a photodiode (PD) or a so-called pinned photo diode (PPD), which is designed such that it deploys during an exposure time of the incident light generates an electrical quantity which represents a measure of the quantity of light received by the relevant pixel. This electrical quantity may be a charge, a voltage, a current or even a time-coded signal, such as a pulse train. Often, such image sensors are constructed as a so-called charge-coupled device (CCD).

In electronic image sensors, such as CCD and CMOS sensors cause disturbances that have no relation to the actual image content (ie the image signal), to a deterioration of the digital or

electronically recorded image. This degradation is called image noise. Such image noise largely consists of the so-called dark noise, which already occurs without light falling on the sensor. The reason for this noise is on the one hand the so-called dark current of the individual photosensitive elements forming the pixels and on the other hand the noise of the readout amplifier (readout noise). Due to the different height of the dark current, individual pixels appear bright on the image, especially at longer exposure times.

In addition, there are also pixels that are constantly bright or dark due to a defect on the sensor chip. In the following, such pixels with deviating brightness values are referred to as defective pixels.

Various solutions have already been proposed for the automatic correction of defect pixels, in principle one can distinguish between methods which use tables and those which work with filters. Nonlinear filters such as

Maximum or median filter used. The table-dependent correction of defect pixels can, for example, be based on tables generated during the production test and stored in the camera. With the filter-dependent correction, it is decided with the help of a filter whether the brightness value of a pixel deviates unusually from the brightness values of its surrounding pixels and, if necessary, replaced on the basis of this. How strong the deviation must be in order to replace a brightness value is usually determined a priori. However, conventional correction methods have the disadvantage that it is not possible to correct a high number of defect pixels at long exposure time and at the same time to change the image as slightly as possible with a short exposure time with few defect pixels. Furthermore, due to a temporary replacement of the brightness value of a pixel between successive images, a continuously changing brightness results, so that the pixel virtually flashes.

The invention has the object of providing a defect pixel correction with the smallest possible change in the image content This object is achieved by a device according to claim 1, a camera according to claim 9, a method according to claim 10 and a computer program according to claim 11.

According to the invention, therefore, a mixture of the pixel values of the original image with those of the low-filtered reference image occurs in the region of the threshold for an admissible brightness deviation, so that a smooth transition function can be achieved in the region of the threshold between correction and non-correction of the pixel information. This transition function avoids blinking of pixels in successive image sequences.

Furthermore, the solution according to the invention is also advantageous in that compared with table-based methods, no individual calibration of each sensor is necessary and a very large number of defect pixels can be corrected.

According to a first advantageous development, the mixing device can be designed such that the mixing ratio within the predetermined range steadily increases with increasing amount of the deviation in favor of the corresponding pixel information of the reference image signal. As a result of this refinement, an admixture of the pixel value of the reference image which increases with the height of the deviation is achieved, so that within the predetermined range, deviating defect pixels are adjusted to a greater degree to the reference image.

According to a second advantageous development, the mixing device can be designed such that the mixing ratio is 50% if the amount of deviation agrees with the predetermined threshold value. This ensures that corrected pixels with a deviation corresponding exactly to the threshold by exactly half mixing of the pixel values of the original image and the reference image.

According to a third advantageous development, the device can be designed such that the pixel information of the original image signal is not corrected when the amount of the deviation is below the predetermined range, and that the pixel information of the original image signal is completely replaced by the pixel information of the reference image signal when the amount of deviation is above the predetermined range. The predetermined range thus forms a transition region for deviations that can not be uniquely assigned to a defect pixel. In this area, a mixture of the pixel value of the original image with that of the reference image takes place according to the invention.

According to a fourth advantageous development, the filter device may comprise a median filter. The median filter is a ranking filter in which the filtering process is based on collecting and sorting the pixel values. Thus, a simple realization with regard to the present data stream is possible.

According to a fifth advantageous development, the mixing device can be configured such that the predetermined

Threshold is changed in response to the result of a comparison of a predetermined expected number with the number of pixels corrected with at least half the mixing ratio. In this case, the predetermined threshold value can preferably be increased if the predetermined expected number is smaller than the number of corrected pixels with at least half the mixing ratio, and reduced if the predetermined expected number is greater than the number of corrected pixels with at least half the mixing ratio. Thus, an automatic adjustment or adjustment of the threshold value is possible in such a way that the image content of the original image is influenced as little as possible.

Further, the apparatus may preferably be configured to calculate the predetermined expected number based on at least one of a sensor constant and a set exposure time. As a result, both the exposure time and the image content can be taken into account in the threshold value adjustment, so that an adaptive defect pixel correction is achieved.

The components of the device or camera proposed for achieving the aforementioned object can be used individually or jointly as discrete circuits, integrated circuits (eg application-specific integrated circuits (ASICs)), programmable circuits (eg Field Programmable Gate Arrays (FPGAs)) or the like , be realized. In particular, the comparison and / or mixing device can be realized by an FPGA. Furthermore, the steps of the subsequent correction method as well as the functionalities of the correction device can be implemented as a software program or software routine for controlling the processor of a computer device for its execution.

In the following, the present invention is explained in more detail by means of preferred embodiments with reference to the drawing figures. Show it:

Fig. 1 is a schematic block diagram of a pixel correction apparatus according to a first embodiment; FIG. 2 is a flow chart of a pixel correction method according to a second embodiment; FIG. and

3 is a more detailed functional diagram of a pixel correction method according to a third embodiment. According to the following embodiments, a camera with modified and thereby more effective pixel correction processing will be described.

1 shows a schematic block diagram with components used in connection with the proposed pixel correction according to a first exemplary embodiment.

In a camera 10, incident light is detected by an image sensor (not shown). As a result, a signal component of the incident light is detected by the image sensor during the opening time (exposure time) of an electrical shutter (not shown). From the incident light, an electrical signal is formed, for example a charge, a voltage, a current or a digital number. When light in the form of photons passes through the camera lens (not shown) onto the image sensor, photo-diodes of the image sensor release electrons which are collected. The more light that is incident, the more electrons are released and the higher the electrical signal. The readout of the individual photodiodes or their charge takes place via an output line, wherein, for example, all the photodiodes of a row are connected to one another and evaluated successively by a read-out register (not shown). For this purpose, the read-out register reads, for example, the charge from the first cell of a line. Subsequently, the remaining charge increases by one cell, so that the first cell of the row is read out again. This creates a data ström from individual analog brightness values, which can be converted, for example, by means of an analog-to-digital converter (A / D converter) into digital data. The pixel values thus obtained determine the brightness of a pixel. Of course, a further processing of the analog pixel values without A / D conversion is possible, wherein the corresponding components of the following embodiments are not configured as digital components but as equivalent to the skilled person readily convertible equivalent analog components.

According to FIG. 1, the pixel values of the original image Bo are fed to a comparator (V) 30 and, in parallel thereto, to a low-pass filter (TP) 20. The low-pass filter 20, for example, reduces the high-frequency signal components of the data stream by averaging, integrating or averaging pixel values in an environment of predetermined neighboring pixels (eg immediate neighboring pixels and / or neighboring second or higher pixels). Thus, the defect pixels leading to unwanted abrupt changes in the data stream are also suppressed. With the help of the low-pass filter so a reference image B _R can be generated, which is largely free of individual defect pixels. This reference image is thus significantly changed due to the low-pass filtering over the original image Bo.

The low-pass filter may be, for example, a non-linear low-pass filter, such as a median filter. This is a so-called ranking filter, which can not be described by a convolution. In the ranking filters, the pixel values are collected in a defined environment of the considered pixel, sorted by size, and ranked. Now it will be Pixel value selected from this sorted list that replaces the pixel value of the current pixel. The choice of position determines the type of ranking filter. For an ascending sort, you get the minimum filter, for the minimum pixel value (first position of the list), the median filter, for the pixel value in the middle of the list, and the maximum filter for the maximum pixel value (last position of the list).

The pixel values of the original image Bo and the reference image B _R are supplied to a comparator 30 and compared pixel by pixel. The comparator 30 determines, for each pixel, the amount of deviation of the pixel values and compares the amount of deviation with a predetermined threshold. Only when the deviation between the pixel value of the original image B ₀ and that of the reference image B _R exceeds the predetermined threshold by a predetermined amount, the pixel value of the original image Bo is replaced with that of the reference image BR. By contrast, if the deviation between the pixel value of the original image B ₀ and that of the reference image BR is below the predetermined threshold by the predetermined amount, the pixel value of the original image Bo is not replaced by that of the reference image B _R and further processed unchanged.

However, if the pixel value of the pixel is near the

Threshold is, the threshold is sometimes exceeded by the ever-present image noise and sometimes not. Due to the resulting continuous change between the pixel value of the original image Bo and that of the reference image B _R , a blinking effect can occur at the relevant pixel. According to the invention, this blinking effect can be avoided by mixing the pixel value of the original image Bo in the vicinity of the predetermined threshold value with that of the reference image B _R by means of a predetermined transition function. To this With the aid of the predetermined transition function (for example a steady-state increasing transition function), the comparator 30 determines a mixing ratio and supplies a control information SI together with the corresponding pixel values of the original image Bo and the reference image B to a mixer (M) 40 which mixes the two pixel values according to the mixing ratio indicated by the supplied control information Sl and outputs a pixel value for a corrected output image B _K. For example, the mixing may be done by a fractional addition or a weighted addition of the pixel values according to the signalized mixing ratio.

The transient function function can be selected and reconciled by simulation or test runs according to the needs of the user and / or the application. For example, the

Transitional function can be chosen so that at the predetermined threshold value a mixing ratio of 50% results, ie a pixel with a pixel value corresponding to the predetermined threshold value is mixed so that the pixel values of the original image B ₀ and of the reference image BR are mixed exactly half.

The correction by means of low-pass filtering (for example median filtering), however, strongly interferes with the image content of the original image, so that particularly finely structured image details can be falsified. In the original image Bo, for example, fine details may be contained which differ only slightly from defect pixels.

Therefore, it is proposed according to an advantageous development of the first embodiment to correct only as many pixels as are to be expected on the basis of the statistical properties of the image sensor (eg CMOS sensor). Thus it is It is advantageous to set the predetermined threshold, which specifies the boundary between normal pixels and defect pixels, as accurately as possible so that exactly the expected number of outliers is corrected. This can be achieved, for example, by counting the pixels corrected to at least 50% (ie with a mixing ratio of at least 50%), for example in the mixer 40, and comparing the result with the number of defect pixels statistically expected for the image sensor used , If it is determined that too many pixels have been corrected, the predetermined threshold value for the next original image Bo is, for example, lowered by a predetermined incremental value. On the other hand, if it is determined that too few pixels have been corrected, the predetermined threshold value is increased by, for example, a predetermined incremental value. For this purpose, the mixer 40 can optionally supply the comparator 30 with, for example, control information S2 signaling the increase or decrease in the predetermined threshold value.

The expected number or specification for the number of pixels to be corrected may be, for example, a sensor constant of

Image sensor and / or the set exposure time of the camera 10 are calculated.

The magnitude of the dark current of the individual pixels of the image sensor is approximately normally distributed, and the dark current is approximately constant. Assuming now an approximately constant threshold, the result is a linear dependence of the expected number of defect pixels on the exposure time according to the following equation: DX (β / tBR) / where n _D denotes the expected number of defect pixels, DR the number of expected defect pixels at a reference exposure time, t _B the currently selected exposure time, and t _B R the reference exposure time. It should be noted that the components or functions of the comparator 30 and of the mixer 40 of the first exemplary embodiment can also be realized integrated in one processing element.

2 shows a flowchart of a pixel correction method according to a second exemplary embodiment, which can be implemented, for example, as a software routine for controlling an image processing processor and can be executed in an endless loop during an image acquisition with the camera for each image in succession. In a first step 200, a corresponding reference image is generated from the original image by means of suitable low-pass filtering. Then, in step 201, the reference image is compared pixel by pixel with the original image. In the following step 202, the pixel values of both images are compared and it is determined for the current pixel whether or not the deviation amount is at least in a predetermined vicinity or closer to a predetermined threshold value. If not, the flow jumps to step 203 and it is determined whether or not the pixel value of the original image has exceeded the predetermined threshold. If it is determined in step 203 that the predetermined threshold has not been exceeded, the process jumps to step 204 and the pixel value of the original image is used without correction for the further image processing and the flow proceeds to step 207. If it is determined in step 203 that that is intended Threshold has been exceeded, the flow jumps to step 205 and the pixel value of the reference image is used for further image processing. Thereafter, the flow advances to step 207. On the other hand, if it is determined in step 202 that the amount of deviation is at least in the predetermined vicinity or closer to the predetermined threshold, the flow advances to step 206 and the pixel values of the original image and the reference image are adjusted their deviation amount.

Then, in step 207, a count value is incremented if the present pixel pixel pixel value has been corrected with at least a predetermined mixing ratio (e.g., 50%). Thereafter, it is checked in step 208 whether all pixels of the current source image have been processed. If not, the process is repeated from step 201.

If it is determined in step 208 that all pixels of the current source image have been processed, the count value last determined in step 207 is compared with an expected default value and it is checked in step 209 if too many pixels have been corrected. If so, flow proceeds to step 210 and the predetermined threshold is lowered. On the other hand, if it is determined in step 209 that not too many pixels have been corrected, it is determined in step 211 whether too few pixels have been corrected. If so, the flow advances to step 212 and the predetermined one

Threshold is increased. On the other hand, if it is determined in step 211 that not too few pixels have been corrected (ie, the count value is in a predetermined range in FIG Near the default value), the corrected image is output in step 213.

Alternatively, in step 200, the reference image may also be generated pixel by pixel. In this case, the process is then repeated from step 208 then from step 200 until all the pixels have been processed.

Finally, FIG. 3 shows a more detailed functional diagram of a pixel correction method according to a third exemplary embodiment. The two square 9-field grids are intended to represent image sensor environments of a pixel marked "X" whose pixel value is processed according to the function shown.

According to FIG. 3, a pixel value x _{0 of} an original image taken by the image sensor is fed to a low-pass filter function 32 in order to obtain a filtered reference pixel value XR ZU. This reference pixel value XR thus corresponds to a background estimate for the source pixel value. The original pixel value xo and the reference pixel value x _R are then subtracted in a subtraction function 33, and the resulting difference value X ₀ -XR is applied to an absolute value function 34, by means of which the positive magnitude value | X ₀ -XR | of the difference value is formed. This absolute value value is then fed to a threshold value function 35 which is continuous and, for example, linearly decreasing within a predetermined range M and a constant output value "1" below the predetermined range M and a constant output value above the predetermined range M. "0" delivers. The output value p of the threshold value function 35 is therefore "0 * for all difference values lying below the predetermined range M and "1" for all difference values above the predetermined range M. For all difference values within the predetermined range M, the output p decreases in proportion to (eg at linear

Function) or at least continuously with increasing difference value amount | XO-XR | , The output value p is thus always in a range between "0" and "1".

Furthermore, according to FIG. 3, the original pixel value xo is multiplied by the reference pixel value x _R in a first multiplication function 36 in order to obtain a value px ₀ . Furthermore, a difference 1-p is formed in a difference-forming function 38, which is then multiplied in a second multiplication function 37 by the reference pixel value XR to obtain a value (lp) x _R. The two values pxo and (1-p) x _R are then added in a final addition function 39 in order to obtain the value px ₀ and (lp) xR as the corrected pixel value.

Thus, it will be appreciated that the proposed pixel correction may be readily implemented in software or hardware by processing according to the functional diagram in FIG. 3, where the function output value p directly indicates the mixing ratio between the original pixel value and the reference pixel value.

Of course, by suitable feedback as a function of the number of corrected pixel values, an adaptive control of the predetermined threshold value S can also be implemented by a corresponding change of the threshold value function 35. The proposed alternative solutions according to the exemplary embodiments can be used for pixel correction in connection with a wide variety of pixel-oriented image acquisition devices or cameras.

In summary, an apparatus and a method for correcting pixel information in one of a

Image sensor Bo obtained with one or two-dimensional pixel arrangement described original image signal Bo, wherein the original image signal B _{0 is} filtered with a low-pass characteristic to produce a reference image signal B _R. The reference picture signal B _R is compared with the original picture signal Bo to detect a deviation between the reference picture signal B _R and the original picture signal B ₀ . Finally, the image point information of the original image signal B _{0 is} then mixed with corresponding pixel information of the reference image signal BR when the deviation between the pixel information of the original image signal Bo and the corresponding pixel information of the reference image signal B _{R is} in a predetermined range around a predetermined threshold.

Claims

claims

Device for correcting pixel information of a defect pixel in an original image signal (Bo) obtained from an image sensor with a one-dimensional or two-dimensional pixel arrangement, wherein the device (21) comprises:

a filter means (20) for filtering the original picture signal (Bo) with a low-pass characteristic to obtain a reference picture signal (B _R );

- Comparing means (30) for comparing the reference image signal (B _R ) with the original image signal (Bo) and for determining a deviation (Sl) between the reference image signal (B _R ) and the original image signal (B ₀ ); and

- Mixing means (40) for mixing the pixel information of the original image signal (Bo) with a corresponding pixel information of the reference image signal (B _R );

if the deviation (Sl) is below a predetermined range (M) by a predetermined threshold value, the original image signal (Bo) is not changed;

if the deviation (Sl) is within the predetermined range (M), the original image signal (B ₀ ) is mixed by the mixer (40) with the reference image signal (B _R ); and - If, if the deviation (Sl) is above the predetermined range (M), the original image signal (Bo) by the reference image signal (B _R ) is replaced.

An apparatus according to claim 1, wherein said mixing means (40) is configured so that the mixing ratio within said predetermined range steadily increases with increasing amount of said deviation (Sl) in favor of the corresponding pixel information of said reference image signal (B _R ).

An apparatus according to claim 1 or 2, wherein the mixing means (40) is configured such that the mixing ratio is 50% when the amount of deviation (Sl) coincides with the predetermined threshold value.

Device according to one of the preceding claims, wherein the filter device comprises a median filter (20).

Apparatus according to any one of the preceding claims, wherein the mixing means (40) is arranged to vary the predetermined threshold value in response to the result of a comparison of a predetermined expected number with the number of corrected pixels having at least half the mixing ratio.

The apparatus of claim 5, wherein the apparatus is configured to increase the predetermined threshold when the predetermined expected number is less than the number of corrected pixels having at least one half of the mixing ratio, and the predetermined one Threshold is reduced if the predetermined expected number is greater than the number of corrected pixels with at least half the mixing ratio.

7. Apparatus according to claim 5 or 6, wherein the apparatus is configured such that the predetermined expected number is calculated from at least one of a sensor constant and a set exposure time.

8. camera (10) with an image sensor and a device according to one of claims 1 to 7.

9. A method of correcting pixel information of a defect pixel in an original image signal (Bo) obtained from a one- or two-dimensional pixel array image sensor, the method comprising the steps of;

- filtering the original picture signal (Bo) with a low-pass characteristic to produce a reference picture signal (B _R );

Comparing the reference picture signal (BR) with the original picture signal (Bo);

Determining a deviation between the reference image signal (B _R ) and the original image signal (Bo);

- Mixing the original image signal (Bo) with the reference image signal (B _R ), if the deviation (Sl) is within a predetermined range (M) by a predetermined threshold value; - maintaining the original image signal (Bo) unchanged if the deviation (Sl) is below the predetermined range (M); and - Replacing the original image signal (Bo) by the reference image signal (BR) if the deviation (Sl) is above the predetermined range (M).

A computer program for storing on a data carrier and for causing a computer device to carry out the control method according to claim 9, when the computer program is executed on the computer device.