CN101448169B - Image processing device and image processing method - Google Patents

Image processing device and image processing method Download PDF

Info

Publication number
CN101448169B
CN101448169B CN2008101823405A CN200810182340A CN101448169B CN 101448169 B CN101448169 B CN 101448169B CN 2008101823405 A CN2008101823405 A CN 2008101823405A CN 200810182340 A CN200810182340 A CN 200810182340A CN 101448169 B CN101448169 B CN 101448169B
Authority
CN
China
Prior art keywords
mentioned
color
image
pixel
planes
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN2008101823405A
Other languages
Chinese (zh)
Other versions
CN101448169A (en
Inventor
沢田保宏
增田孝
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Acutelogic Corp
Original Assignee
Acutelogic Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Acutelogic Corp filed Critical Acutelogic Corp
Publication of CN101448169A publication Critical patent/CN101448169A/en
Application granted granted Critical
Publication of CN101448169B publication Critical patent/CN101448169B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H04N23/843—Demosaicing, e.g. interpolating colour pixel values
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/12—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths with one sensor only
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2101/00—Still video cameras

Abstract

The invention provides an image processing device and an image processing method capable of saving a memory capacity of a buffer or a band so that low power consumption and low costs can be realized when a color image is generated from a mosaic image outputted from an image picking-up element with a single plate color filter and an image modification processing is carried out. An image processing device is provided with a buffer memory (21) that stores a color mosaic image outputted from an image picking-up element (5) and a demosaic unit (28) that carries out an image modification for the color mosaic image outputted from the buffer memory (21) and generates a color image which has a plurality of color information data for every pixel.

Description

Image processing apparatus and image processing method
Technical field
The present invention relates to the image processing apparatus and the image processing method of the shot object image that the image unit by electronics photographs, relate in particular to via the imaging apparatus with colour filter of one-board and export, and by each pixel by the mosaic image of the monochrome information that only has solid color be created on each pixel have a plurality of colors monochrome information coloured image and implement the camera head that anamorphose is handled.
Background technology
In the past, known in camera heads such as digital camera, by camera lens shot object image is imaged on the imaging apparatus, and generates the image processing apparatus and the image processing method of picture signal by means of this imaging apparatus carries out light-to-current inversion to shot object image.
In addition, also have following image processing apparatus and image processing method, promptly, imaging apparatus as one-board, be a plurality of components of photo-electric conversion of rectangular formation, and, have each colour filter of R (red) G (green) B (indigo plant) accordingly at its place ahead and the components of photo-electric conversion, the picture signal of the solid color exported via this colour filter is applied signal processing and generates coloured image.
That is, the image of exporting via the imaging apparatus of one-board is the color mosaic image of each pixel colouring information of only having solid color, in order to generate coloured image, needs each pixel to have redness (R), green (G), blue a plurality of colouring informations such as (B).
Therefore, in the image processing of having used the one-board imaging apparatus, only have any one the color mosaic image of colouring information in R, G, the B composition, carry out so-called anti-mosaic processing (being also referred to as color interpolation handles) based on each pixel.At this, anti-mosaic processing is to use other color of pixel information of its pixel periphery to carry out interpolation arithmetic by other colouring information that will lack in each pixel of color mosaic image, generates the processing (so-called color interpolation processing) of coloured image that each pixel has all colours information of R, G, B composition respectively.
In addition, in the image processing of having utilized the one-board imaging apparatus, often after having carried out anti-mosaic processing by the color mosaic image as described above, carry out rocking and crooked grade or the anamorphose processing of zoomed image as required of correcting image.
That is, in camera heads such as digital camera, when taking above-mentioned subject, digital camera is run counter to user's wish and is shaken the image confusion that causes, the so-called crooked problem that becomes of rocking and being caused by pick-up lens.Therefore, known Digital Signal Processing by gyroscope and image detects shaking of camera, moves or makes the picture signal electronics move (being the correction pixels position) by the optical axis optics that makes camera lens and proofread and correct and rock and crooked technology.
At this moment,, not only carry out parallel moving, also be rotated and tractive (ア オ リ) the such anamorphose and the position correction of sub-pixel unit in order to obtain more high-quality picture element.
For example, when the image after the output image of imageing sensor extracts the distortion of having carried out some rotations and convergent-divergent is exported shown in Fig. 9 (a), usually require to read from imageing sensor by raster scan, the image after the distortion also outputs to the back level by raster scan.At this, during first row after output image distortion, need can be with reference to the ground buffer memory with the presumptive area (shadow region) of imageing sensor output image.In addition, in the rotations of carrying out 90 degree with when anamorphose such as spinning upside down, this presumptive area becomes the amount of 1 picture.And, installing, owing to be the structure that makes the Double Register that separates as the frame that writes object with as the frame of reading object, so the image buffer (for example, with reference to patent documentation 1,2) of the 2 frame amounts of needs.
In detail, shown in Fig. 9 (b), video camera with " from the one-board color image sensor output HD motion video of 5M pixel " is an example, in existing anti-mosaic processing, in order to generate coloured image accordingly in each pixel of Baeyer arrangement and each pixel of output image, at first, generate the RGB coloured image (being the coloured image that each pixel has a plurality of colouring informations) of 5M by 5M color of pixel mosaic image from imageing sensor output, next, to the such anamorphose of change that this RGB coloured image carries out jitter correction and picture size, the coloured image of output 2M.
At this moment, need will become the RGB coloured image buffer memory of the 5M of anamorphose object, be 24bit/pix so get the RGB data, and when being taken as Double Register for motion video handles, the required memory capacity of buffer storage is 240Mbit (2 * 5M * 24).
In addition, when considering the processing speed of 60fps, even with reference to each pixel respectively once, the required frequency band of this buffer storage just is 14.4Gbps (2 (amount that writes and read) * 5M * 24 * 60).In addition, because in fact sometimes with reference to repeatedly, so the required frequency band of buffer storage will be on this.
In addition, as the compress technique of coloured image, known have pair output of one-board color image sensor not carry out color interpolation to handle and directly compress technology (for example, referring to Patent Document 3) to dwindle hardware.
[patent document 1] TOHKEMY 2007-228515 communique
[patent document 2] TOHKEMY 2007-228514 communique
[patent document 3] TOHKEMY 2005-217896 communique
But, in the image processing apparatus that has utilized the one-board imageing sensor, coloured image (3parm/pix) was carried out anamorphose in the past always, so buffer needs big capacity and frequency band, have the danger that power consumption increases and cost increases takes place.In addition, the technology of patent document 3 discloses the technology of the mosaic image before the generation coloured image being compressed for the miniaturization of seeking hardware, but the technology when this technology of public use is not carried out anamorphose effectively and generated coloured image, comprise in anamorphose and color be created on, the leeway of invention is still arranged.
Summary of the invention
Therefore, the objective of the invention is to, a kind of image processing apparatus and image processing method are provided, generate by color mosaic image from one-board imaging apparatus output with colour filter each pixel have a plurality of colors monochrome information coloured image and when carrying out anamorphose and handling, the memory capacity and the frequency band of buffer be can save, low power consumption and cost degradation realized.
The technical scheme 1 described invention of making in order to achieve the above object is a kind of image processing apparatus, its color mosaic image that has a monochromatic monochrome information by each pixel generates the monochrome information of a plurality of colors in each pixel, and implement the predetermined picture distortion and generate coloured image, wherein, above-mentioned color mosaic image is to be obtained by the imaging apparatus of one-board that different a plurality of color of light are carried out the pixel of light-to-current inversion that has in the image unit
Above-mentioned image processing apparatus comprises: planes of color decomposition portion, and it resolves into above-mentioned color mosaic image a plurality of planes of color of the monochrome information that only comprises same color of light; Buffer storage, the color mosaic image after its each planes of color storage according to above-mentioned a plurality of planes of color is separated by above-mentioned planes of color decomposition part; The camera lens state of the pick-up lens in the aberration coefficients table, itself and above-mentioned image unit records the distortion aberration coefficients of the distortion aberration that is used to proofread and correct described imaging apparatus accordingly; Rock test section, it rocks the corrected value that rock of calculated signals with respect to the color mosaic image of described image unit input according to the output of angular-rate sensor from described image unit; And anti-mosaic portion, it generates having implemented the above-mentioned coloured image after the predetermined picture distortion from the color mosaic image of above-mentioned buffer storage output,
Above-mentioned anti-mosaic portion comprises output pixel scanner section, Coordinate Conversion portion, sampling portion, interpolating portion and color generating unit, the such anamorphose of convergent-divergent of unified various aberration corrections of above-mentioned output pixel scanner section and jitter correction, image, and the pixel coordinate of the coloured image of output scanned; Above-mentioned Coordinate Conversion portion rocks the signal that test section and above-mentioned aberration coefficients table are exported according to expression from the signal of the pixel coordinate of above-mentioned output pixel scanner section output with from above-mentioned, calculate the sample coordinate on the above-mentioned color mosaic image corresponding with the location of pixels of the above-mentioned coloured image of exporting, above-mentioned interpolating portion is according to each planes of color, according to the pixel value that obtains by above-mentioned sampling portion, interpolation generates each color pixel value under the sample coordinate, above-mentioned color generating unit is synthesized each color pixel value under the sample coordinate that is obtained by above-mentioned interpolating portion, generates the color data that each pixel has a plurality of color components.
According to technical scheme 1 described image processing apparatus, above-mentioned image processing apparatus comprises: planes of color decomposition portion, and it resolves into above-mentioned color mosaic image a plurality of planes of color of the monochrome information that only comprises same color of light; Buffer storage, the color mosaic image after its each planes of color storage according to above-mentioned a plurality of planes of color is separated by above-mentioned planes of color decomposition part; And anti-mosaic portion, it generates implemented the above-mentioned coloured image after the predetermined picture distortion from the color mosaic image of above-mentioned buffer storage output, above-mentioned anti-mosaic portion comprises the output pixel scanner section, Coordinate Conversion portion, sampling portion, interpolating portion and color generating unit, above-mentioned Coordinate Conversion portion is according to the signal of expression from the pixel coordinate of above-mentioned output pixel scanner section output, calculate the sample coordinate on the above-mentioned color mosaic image corresponding with the location of pixels of the above-mentioned coloured image of exporting, above-mentioned interpolating portion is according to each planes of color, according to the pixel value that obtains by above-mentioned sampling portion, interpolation generates each color pixel value under the sample coordinate, above-mentioned color generating unit is synthesized each color pixel value under the sample coordinate that is obtained by above-mentioned interpolating portion, generate the color data that each pixel has a plurality of color components, therefore, generate by color mosaic image from one-board imaging apparatus output each pixel have a plurality of colors monochrome information coloured image and when carrying out anamorphose and handling, with coloured image carried out anamorphose compare, the memory capacity and the frequency band of buffer be can save, low power consumption and cost degradation realized.Promptly, according to the structure that coloured image is stored in the buffer, per 1 pixel need with corresponding 3 variablees of RGB, but by as the present invention, the color mosaic image being stored in the buffer storage, the variable that can make per 1 pixel is 1, can provide to save the required capacity of buffer storage and frequency band and realize low power consumption and cost degradation and the camera head with anamorphose function.In addition, in technical scheme 1 described image processing apparatus, in the sample coordinate on the color mosaic image of each planes of color that the location of pixels with coloured image calculates accordingly, also reflected anamorphose with anti-mosaic (generation of coloured image), therefore, compare with the anamorphose processing with carrying out anti-mosaic processing respectively, can alleviate and handle load.
For example, as shown in Figure 3, be example with the video camera of " from the one-board color image sensor of 5M pixel output HD motion video ", at this moment,, the color mosaic image is carried out anamorphose and anti-mosaic processing with color mosaic image buffer memory.At this moment, the color mosaic image is 12bit/pix, and when being taken as Double Register for motion video handles, the required memory capacity of buffer storage is 120Mbit (2 * 5M * 12), compares with existing example shown in Figure 9 memory capacity is reduced by half.
In addition, technical scheme 1 described image processing apparatus as technical scheme 2 described inventions, also comprises compression unit, and its compression is from the above-mentioned color mosaic image of above-mentioned imaging apparatus; And extension section, it stretches by the above-mentioned color mosaic image that is stored in after the above-mentioned compressing section compresses in the above-mentioned buffer storage, above-mentioned compression unit is divided into a plurality of according to each planes of color of above-mentioned a plurality of planes of color, be compressed into below the pre-sizing according to this each piece that is split to form, the color mosaic image of each piece of above-mentioned buffer memory stores after by above-mentioned compressing section compresses, above-mentioned extension section is according to each piece that is stored in the above-mentioned buffer storage, color mosaic image after the Compression and Expansion, above-mentioned anti-mosaic portion generates above-mentioned coloured image according to the color mosaic image after being stretched by above-mentioned extension section.Thus, can further cut down the memory capacity of buffer storage.In addition, can be easily will compress accordingly, and the color mosaic image after can Compression and Expansion generates coloured image from color mosaic image and each color of the output of one-board color image sensor.
In addition, technical scheme 2 described image processing apparatus, preferred above-mentioned compression unit carry out and the corresponding gray scale compression of the gray scale of above-mentioned brightness.
In addition, technical scheme 1 or 2 described image processing apparatus, the preferred cache memory that also comprises the pixel value of storing above-mentioned color mosaic image, above-mentioned anti-mosaic portion can not obtain above-mentioned pixel value via above-mentioned buffer storage from above-mentioned cache memory.That is, in anti-mosaic processing, when generating 1 pixel of coloured image, with reference to a plurality of color pixel values in the color mosaic image, but when generating the adjacent pixels of coloured image, many same pixels with reference to (visit) color mosaic image.Therefore, the pixel by reference just before making it is stored in the cache memory, can cut down the frequency band of reading of buffer storage, further Xiao Minus power consumption.
And, the pixel value group in the pixel value of the pixel that preferred above-mentioned cache memories store was visited recently or above-mentioned.
Next, technical scheme 3 described inventions are a kind of image processing methods, the color mosaic image that has a monochromatic monochrome information by each pixel generates the monochrome information of a plurality of colors in each pixel, and implement the predetermined picture distortion and generate coloured image, wherein, above-mentioned color mosaic image is to be obtained by the imaging apparatus of one-board that different a plurality of color of light are carried out the pixel of light-to-current inversion that has in the image unit
Above-mentioned image processing method comprises: the planes of color decomposition step, resolve into above-mentioned color mosaic image a plurality of planes of color of the monochrome information that only comprises same color of light; The planes of color storing step is stored in color mosaic image after decomposing in the above-mentioned planes of color decomposition step according to each planes of color of above-mentioned a plurality of planes of color in buffer storage; And anti-mosaic step, generate having implemented the above-mentioned coloured image after the predetermined picture distortion from the color mosaic image of above-mentioned buffer storage output,
Above-mentioned anti-mosaic step comprises that output pixel scanning step, Coordinate Conversion step, sampling step, interpolation procedure and color generate step, in above-mentioned output pixel scanning step, the such anamorphose of convergent-divergent of unified various aberration corrections and jitter correction, image, and the pixel coordinate of the coloured image of output scanned; In above-mentioned Coordinate Conversion step, according to the signal that is illustrated in the pixel coordinate of exporting in the above-mentioned output pixel scanning step with from rocking the signal of test section and aberration coefficients table output, calculate the sample coordinate on the above-mentioned color mosaic image corresponding with the location of pixels of the above-mentioned coloured image of exporting, wherein, the above-mentioned test section that rocks rocks the corrected value that rock of calculated signals with respect to the color mosaic image of above-mentioned image unit input according to the output of angular-rate sensor from above-mentioned image unit; The camera lens state of the pick-up lens in above-mentioned aberration coefficients table and the above-mentioned image unit records the distortion aberration coefficients of the distortion aberration that is used to proofread and correct above-mentioned imaging apparatus accordingly; In above-mentioned interpolation procedure, according to each planes of color, according to the pixel value that in above-mentioned sampling step, obtains, interpolation generates each color pixel value under the sample coordinate, generate in the step in above-mentioned color, each color pixel value under the sample coordinate that obtains in above-mentioned interpolation procedure is synthesized, generate the color data that each pixel has a plurality of color components.
According to technical scheme 3 described image processing methods, above-mentioned image processing method comprises: the planes of color decomposition step, resolve into above-mentioned color mosaic image a plurality of planes of color of the monochrome information that only comprises same color of light; The planes of color storing step is stored in color mosaic image after decomposing in the above-mentioned planes of color decomposition step according to each planes of color of above-mentioned a plurality of planes of color in buffer storage; And anti-mosaic step, generation is to having implemented the above-mentioned coloured image after the predetermined picture distortion from the color mosaic image of above-mentioned buffer storage output, above-mentioned anti-mosaic step comprises the output pixel scanning step, the Coordinate Conversion step, sampling step, interpolation procedure and color generate step, in above-mentioned Coordinate Conversion step, according to the signal that is illustrated in the pixel coordinate of exporting in the above-mentioned output pixel scanning step, calculate the sample coordinate on the above-mentioned color mosaic image corresponding with the location of pixels of the above-mentioned coloured image of exporting, in above-mentioned interpolation procedure, according to each planes of color, according to the pixel value that in above-mentioned sampling step, obtains, interpolation generates each color pixel value under the sample coordinate, generate in the step in above-mentioned color, each color pixel value under the sample coordinate that obtains in above-mentioned interpolation procedure is synthesized, generate the color data that each pixel has a plurality of color components, therefore, identical with technical scheme 1 described invention, with coloured image carried out anamorphose compare, the memory capacity and the frequency band of buffer be can save, low power consumption and cost degradation realized.In addition, in technical scheme 3 described image processing methods, in the sample coordinate on the color mosaic image of each planes of color that the location of pixels with coloured image calculates accordingly, also reflected anamorphose with anti-mosaic (generation of coloured image), therefore, compare with the anamorphose processing with carrying out anti-mosaic processing respectively, can alleviate and handle load.
In addition, technical scheme 3 described image processing methods as technical scheme 4 described inventions, also comprise compression step, and compression is from the above-mentioned color mosaic image of above-mentioned imaging apparatus; And stretching step, be stretching in the above-mentioned color mosaic image that is stored in after the compression in the above-mentioned compression step in the above-mentioned buffer storage, in above-mentioned compression step, each planes of color according to above-mentioned a plurality of planes of color is divided into a plurality of, be compressed into below the pre-sizing according to this each piece that is split to form, in above-mentioned planes of color storing step, in above-mentioned buffer storage, be stored in the color mosaic image of each piece after the compression in the above-mentioned compression step, in above-mentioned stretching step, according to each piece that is stored in the above-mentioned buffer storage, color mosaic image after the Compression and Expansion, in above-mentioned anti-mosaic step, generate above-mentioned coloured image according to the color mosaic image after in above-mentioned stretching step, stretching.Thus, identical with technical scheme 2 described inventions, can further cut down the memory capacity of buffer storage.In addition, can be easily will compress accordingly, and the color mosaic image after can Compression and Expansion generates coloured image from color mosaic image and each color of the output of one-board color image sensor.
In addition, technical scheme 4 described image processing methods preferably in above-mentioned compression step, carry out and the corresponding gray scale compression of the gray scale of above-mentioned brightness.
In addition, technical scheme 3 or 4 described image processing methods, the preferred cache memory that uses the pixel value of the above-mentioned color mosaic image of storage in above-mentioned anti-mosaic step, can not obtain above-mentioned pixel value via above-mentioned buffer storage from above-mentioned cache memory.Thus, in anti-mosaic processing, when generating 1 pixel of coloured image, with reference to a plurality of color pixel values in the color mosaic image, but when generating the adjacent pixels of coloured image, many same pixels with reference to (visit) color mosaic image.Therefore, the pixel by reference just before making it is stored in the cache memory, can cut down the frequency band of reading of buffer storage, further Xiao Minus power consumption.
And, the pixel value group in the pixel value of the pixel that preferred above-mentioned cache memories store was visited recently or above-mentioned.
Image processing apparatus of the present invention and image processing method resolve into the color mosaic image a plurality of planes of color of the monochrome information that only comprises same color of light, each planes of color according to a plurality of planes of color is stored the color mosaic image in buffer storage, then, each planes of color according to a plurality of planes of color that are stored in the color mosaic image in the buffer storage, calculate the sample coordinate on the corresponding color mosaic image of the location of pixels of the coloured image when having implemented anamorphose, then, each planes of color according to a plurality of planes of color that are stored in the color mosaic image in the buffer storage, pixel value according to the same color of light that comprises in the planes of color, each color pixel value under the sample coordinate on the interpolation generation color mosaic image, then, each color pixel value under the sample coordinate of interpolation generation is synthesized, generate coloured image, therefore, generate by color mosaic image from one-board imaging apparatus output each pixel have a plurality of colors monochrome information coloured image and when carrying out anamorphose and handling, with coloured image carried out anamorphose compare, the memory capacity and the frequency band of buffer be can save, low power consumption and cost degradation realized.Promptly, according to the structure that coloured image is stored in the buffer, per 1 pixel need with corresponding 3 variablees of RGB, but by as the present invention, the color mosaic image being stored in the buffer storage, the variable that can make per 1 pixel is 1, can provide to save the required capacity of buffer storage and frequency band and realize low power consumption and cost degradation and the camera head with anamorphose function.In addition, in the sample coordinate on the color mosaic image of each planes of color that the location of pixels with coloured image calculates accordingly, also reflected anamorphose with anti-mosaic (generation of coloured image), therefore, compare with the anamorphose processing with carrying out anti-mosaic processing respectively, can alleviate and handle load.
In addition, image processing apparatus of the present invention and image processing method are divided into a plurality of according to each planes of color of a plurality of planes of color, be compressed into below the pre-sizing according to this each piece that is split to form, then, the color mosaic image of each piece in buffer storage after the store compressed, then, according to each piece that is stored in the buffer storage, color mosaic image after the Compression and Expansion, generate coloured image according to the color mosaic image after stretching, thus, can further cut down the memory capacity of buffer storage.In addition, can be easily will compress accordingly, and the color mosaic image after can Compression and Expansion generates coloured image from color mosaic image and each color of the output of one-board color image sensor.
Description of drawings
Fig. 1 is the block diagram that the structure of camera head 1 image processing apparatus of the present invention and image processing method, first execution mode has been used in expression.
Fig. 2 is the planes of color decomposition portion in this first execution mode and the function declaration figure of color generating unit, the figure of the color mosaic image that (a) to be expression arrange from the Baeyer of image unit 2 output, (b), the key diagram when (c), (d), (e) are the figure of the configuration on the R plane that generated by planes of color decomposition portion of expression, Gr plane, Gb plane, B plane, pixel value that (f) is the interpolation calculation sample coordinate respectively.
Fig. 3 is the key diagram that carries out the anti-mosaic processing of anamorphose and generation coloured image in this first execution mode, by the color mosaic image.
Fig. 4 is the image processing apparatus of this first execution mode of expression and the flow chart of the order that the coloured image in the image processing method generates.
Fig. 5 is the block diagram that the structure of camera head 1A image processing apparatus of the present invention and image processing method, second execution mode has been used in expression.
Fig. 6 is the image processing apparatus of this second execution mode of expression and the flow chart of the order that the coloured image in the image processing method generates.
Fig. 7 is the flow chart of details of the order of compression in the presentation graphs 6 and speed buffering controller.
Fig. 8 is the figure from cache memory scanning example during by each piece output pixel value of expression in the variation.
Fig. 9 is the key diagram that existing anti-mosaic processing and anamorphose are handled.
1,1A: camera head; 2: image unit; 3: pick-up lens; 5: imaging apparatus; 5a: the colour filter that Baeyer is arranged; 6:AFE (Analog Front End); 7: correlated double sampling circuit; 8: variable gain amplifier (AGC:Automatic Gain Control); The 9:A/D transducer; 13:TG (Timing Generator); 15: angular-rate sensor; 18:CPU (Central Processing Unit); 19:ROM (Read Only Memory); 20: compression unit; 21: buffer storage; The 22:R flat memory; The 23a:Gr flat memory; The 23b:Gb flat memory; The 24:B flat memory; 25: extension section; 26: the speed buffering controller; 27: cache memory; 28: anti-mosaic portion; 29: the output pixel scanner section; 30: Coordinate Conversion portion; 31: sampling portion; 32: interpolating portion; 33: the color generating unit; 34: vision correction portion; 35: compression unit; 36: recording portion; 38: the aberration coefficients table; 40: rock test section; 100,100A: image processing apparatus.
Embodiment
(first execution mode)
Next, based on first execution mode of description of drawings image processing apparatus of the present invention and image processing method.
Fig. 1 is the block diagram that the structure of camera head 1 image processing apparatus of the present invention and image processing method, first execution mode has been used in expression, Fig. 2 is the planes of color decomposition portion in this first execution mode and the function declaration figure of color generating unit, (a) be the figure of expression from the color mosaic image of the Baeyer arrangement of image unit 2 outputs, (b), (c), (d), (e) be respectively the figure of the configuration on the R plane that generated by planes of color decomposition portion of expression, Gr plane, Gb plane, B plane, the key diagram when (f) being the pixel value of interpolation calculation sample coordinate.
In addition, Fig. 3 is in this first execution mode, carried out anamorphose and generated the key diagram of the anti-mosaic processing of coloured image by the color mosaic image, and Fig. 4 is the image processing apparatus of this first execution mode of expression and the flow chart of the order that the coloured image in the image processing method generates.
Camera head 1 is the camera module that for example carries on the mobile device such as mobile phone, as shown in Figure 1, constitute by image unit 2 and image processing apparatus 100, wherein, above-mentioned image unit 2 guides to imaging apparatus 5 with shot object image P and exports as data image signal (picture signal of mosaic shape), above-mentioned image processing apparatus 100 is according to the data image signal via image unit 2 outputs, carry out rolling dynamic(al) correction and the predetermined picture distortion of image unit 2, and generate the coloured image that each pixel has a plurality of colouring informations with respect to subject.In addition, the camera head 1 of present embodiment in image processing apparatus 100, will store buffer storage 21 described later into from the color mosaic image of image unit 2 output as shown in Figure 3, accesses buffer 21 and carry out anamorphose and anti-mosaic processing.
Next, as shown in Figure 1, image unit 2 comprises: the pick-up lens 3 that shot object image P is guided to imaging apparatus 5, the imaging apparatus (CCD:Charge Coupled Devices) 5 that the shooting light that receives is converted to electrical quantities and exports, the AFE (Analog Front End) 6 that will be converted to data image signal C from the analog picture signal of imaging apparatus 5 outputs and export, with predetermined periodic Control imaging apparatus 5 and the TG (Timing Generator) 13 of AFE6, and detect camera head 1 and rock and export with the angular-rate sensor 15 of the corresponding signal of telecommunication of its rolling momentum etc. with respect to subject.
Imaging apparatus 5 is one-board imaging apparatuss, it constitutes a plurality of components of photo-electric conversion and is rectangular configuration, have accordingly by the trichromatic Baeyer of R (red) G (green) B (indigo plant) (Bayer) at its front and the components of photo-electric conversion and to arrange the colour filter 5a that constitutes, the light quantity of the solid color of color filter portions that will be by each color is converted to analog electrical signal.In addition, export analog electrical signal from imaging apparatus 5 with raster order.
Above-mentioned primary colors Baeyer is arranged as Fig. 2 (a) shown in, and the colour filter of G look disposes with grid pattern, and the row of G look colour filter and R look colour filter alternate configurations and the row that disposed G colour filter and B colour filter replace formation.In addition, in the present embodiment, in Fig. 2 (a), will be disposed at a G color table on the direction side by side with R and be shown Gr, will be disposed at a G color table on the direction side by side with B and be shown Gb.
AFE6 by the correlated double sampling circuit (CDS:Correlated Double Sampling) 7 that the analog picture signal via imaging apparatus 5 output is removed denoising, will carry out the variable gain amplifier (AGC:Automatic Gain Control) 8 that the picture signal of correlated-double-sampling amplifies and will be converted to the formations such as A/D converter 9 of data image signal by correlated double sampling circuit 7 via the analog picture signal of variable gain amplifier 8 inputs, will be converted to data image signal C and output to image processing apparatus 100 from the picture signal of imaging apparatus 5 outputs.
In addition, in image unit 2, also can use CMOS (Complementary Metal Oxide Semiconductor) transducer to replace imaging apparatus 5, correlated double sampling circuit 7, variable gain amplifier 8 and A/D converter 9 etc.Because only have the information of solid color from the signal of each pixel of imaging apparatus 5 output, so from the picture signal of image unit 2 to image processing apparatus 100 output mosaic shapes.In addition, just be also referred to as the RAW data by the color mosaic image after the AD conversion.
Angular-rate sensor 15 for example is made of known vibratory gyroscope, generate with this image unit 2, to the rocking of the rocking of the rocking of subject fore-and-aft direction, left and right directions, above-below direction, be that the rotation of rotating shaft is rocked, is that the rotation of rotating shaft is rocked, the corresponding signal of telecommunication (hereinafter referred to as rocking signal) such as rocked and output to image processing apparatus 100 to the rotation that is rotating shaft with upper and lower with the left and right directions with the fore-and-aft direction.In addition, also can use the 3-axis acceleration sensor of piezo-electric type or use a plurality of acceleration transducers to replace angular-rate sensor 15 accordingly with orthogonal 3 direction of principal axis.In addition, when detection is rocked, also can in image processing apparatus 100, detect the difference of the picture signal between a plurality of images of taking in the given time and detect above-mentioned rocking, replace utilizing the sensor to detect and rock according to these differences.
Next, image processing apparatus 100 is by preserving (storage) buffer storage 21 from the mosaic image of image unit 2 outputs; Generation is to the anti-mosaic portion 28 of the coloured image of having implemented the predetermined picture distortion from the color mosaic image of buffering memory 21 output; The vision correction portion 34 that color picture signal from 33 outputs of color generating unit is carried out known Gamma correction and chromaticity correction, edge enhancing etc. for the visual quality that improves color image; The compression unit 35 that will compress with methods such as for example JPEG via the coloured image of vision correction portion 34 outputs; To record the recording portion 36 in the recording medium such as flash memory for example via the coloured image of compression unit 35 output; CPU (Central Processing Unit) 18; And formation such as ROM (Read Only Memory) 19, CPU18 controls the various processing of this camera head 1 and image processing apparatus 100 according to the control program that is stored among the ROM19.
In addition, image processing apparatus 100 comprises: record the aberration coefficients table 38 of the distortion aberration coefficients of the distortion aberration that is used to proofread and correct image unit 2 accordingly with the camera lens state of pick-up lens 3, according to rocking calculated signals with respect to rocking test section 40 etc. from the corrected value that rocks of the color mosaic image of image unit 2 inputs from angular-rate sensor 15 outputs.
Buffer storage 21 is by realizations such as DRAM, arrange accordingly with Baeyer, constitute by the Gb flat memory 23b of the picture element signal of Gr flat memory 23a, the storage Gb of the picture element signal of the R flat memory 22 of the picture element signal of storage R, storage Gr, the B flat memory 24 of picture element signal of storage B, R pixel, Gr pixel, Gb pixel, B pixel are preserved according to the order of sequence with raster order respectively, according to instruction, these picture element signals (hereinafter referred to as pixel value) are outputed to sampling portion 31 in the anti-mosaic portion 28 from CPU18.Planes of color decomposition portion among the present invention realizes its function by means of buffer storage 21.
In aberration coefficients table 38, preserve the aberration coefficients of the aberration that expression causes by pick-up lens 3.This aberration coefficients is output to the coordinate converting section 30 in the anti-mosaic portion 28.
Rock test section 40 according to the signal of telecommunication that rocks from angular transducer 15 outputs, detect image unit 2 with respect to the rocking of subject, will be used for proofreading and correct the coordinate converting section 30 that correction parameter (z, dx, dy, θ, ψ y, ψ x etc.) outputs to anti-mosaic portion 28 of rocking of rocking.
In rocking correction parameter (below be also referred to as rock corrected value), for subject, z is the corrected value with the size of rocking shot object image together of the fore-and-aft direction of image unit 2, and dx is with left and right directions or waves the corrected value of the position, the left and right sides of rocking shot object image together of axle, dy is the corrected value with the upper-lower position that rocks shot object image together of above-below direction or pitch axis, θ is and the corrected value that with the fore-and-aft direction is the rotation of rocking shot object image together of rotating shaft, ψ y is the corrected value with the tractive of the above-below direction that rocks shot object image together of above-below direction or pitch axis, ψ x is with left and right directions or waves the corrected value of tractive of the left and right directions that rocks shot object image together of axle.In addition, rock the zoom ratio that correction parameter z also can comprise camera head 1.
Next, anti-mosaic portion 28 comprises: the such anamorphose of convergent-divergent of unified various aberration corrections and jitter correction, image, and the output pixel scanner section 29 that the pixel coordinate of the coloured image of output is scanned; From the signal of the pixel coordinate of output pixel scanner section 29 output with from rocking the signal of test section 40 and the output of aberration coefficients table 38, calculate the coordinate converting section 30 of the sample coordinate on the color mosaic image corresponding according to expression with the location of pixels of the coloured image of exporting; The pixel value of each color pixel near the color mosaic image the sample coordinate that will be calculated by coordinate converting section 30 is from the sampling portion 31 that buffering memory 21 reads in; Generate the interpolating portion 32 of the pixel value of sample coordinate by the pixel value interpolation that obtains by sampling portion 31 by the planes of color of each R, Gr, Gb, B; And each color pixel value of the synthetic sample coordinate that obtains by interpolating portion 32, generate the color generating unit 33 etc. that each pixel has the color data of a plurality of color components.
In addition, when carrying out above-mentioned anamorphose, because the location of pixels of color mosaic image and output coloured image is not merely corresponding, so set up its corresponding relation by the coordinate converting section 30 in this anti-mosaic portion 28.Next, the details that anti-mosaic portion 28 handles is described.
In output pixel scanner section 29, with whole pixels of the coloured image of raster order scanning output, and with the coordinate (u of each pixel d, v d) output to coordinate converting section 30.
Next, coordinate converting section 30 bases are from rocking the aberration coefficients of preserving correction parameter and the aberration coefficients table 38 of rocking of test section 40 outputs, comprise and distort the such anamorphose of aberration correction and jitter correction, digital zoom, calculate location of pixels (u with the coloured image of output interior d, v d) sample coordinate on the corresponding color mosaic image.In addition, the anamorphose among the present invention realizes its function by means of coordinate converting section 30.
At first, as coordinate system, will represent that with uv the coordinate system that will be convenient to use is represented with xy in correction calculation with the corresponding coordinate system of location of pixels.In addition, subscript s is illustrated in the coordinate on the color mosaic image, and subscript d is illustrated in the coordinate on the output coloured image.
In addition, to the coloured image both sides, the xy coordinate system after the introducing normalization.In the xy coordinate system, be initial point with the picture centre, making its image is 2 to angular length.
That is, if make coloured image be of a size of 1920 * 1080 pixels, the location of pixels (u of Shu Chu coloured image then d, v d) showing as under the xy coordinate system (formula 1).That is, when the equally spaced pixel that is of a size of 1920 * 1080 by coloured image is arranged formation, calculate (x y)=(960/1100,540/1100)=(0.87,0.49), makes picture centre (u to give skew (offset) d, v d)=(960,540) be initial point (x d, y d)=(0,0), obtain (formula 1).
[formula 1]
x d y d = U d / 1100 - 0.87 V d / 1100 - 0.49 Formula 1
Next, use the parameter (z, θ, dx, dy, ψ x, ψ y) of above-mentioned rolling dynamic(al) correction, the coordinate (x after the rolling dynamic(al) correction is calculated in utilization (formula 2) s, y s).In addition, in (formula 2), establish x s=wx s/ w, y s=wy s/ w.
[formula 2]
wx s wy s w = z cos θ z sin θ dx - z sin θ z cos θ dy ψx ψy 1 x d y d 1 Formula 2
Rock test section 40 and also can output to coordinate converting section 30, replace to rock correction parameter (z, dx, dy, θ, ψ y, ψ x etc.) and output to coordinate converting section 30 containing this matrix that rocks (formula 2) of correction parameter itself.And coordinate converting section 30 is directly used these matrixes itself and is calculated the coordinate (x that shakes after the dynamic(al) correction s, y s) get final product.
Further, the coordinate after the rolling dynamic(al) correction is used the coefficient k that is stored in the distortion aberration correction in the aberration coefficients table 38 1, k 2, p 1, p 2, use arithmetic expression x SG=x ' (1+k 1R ' 2+ k 2R ' 4)+2p 1X ' y '+p 2(r ' 2+ 2x ' 2), y SG=y ' (1+k 1R ' 2+ k 2R ' 4)+2p 2X ' y '+p 1(r ' 2+ 2y ' 2) calculate the Gr plane be stored in the buffer storage 21 and the sample coordinate x in the Gb plane SG, y SGAt this moment, establish r ' 2≡ x ' 2+ y ' 2
k 1, k 2, p 1, p 2Be the coefficient of the distortion aberration of expression pick-up lens 3, k 1, k 2Crooked, the p of expression radioactive ray direction 1, p 2Represent the crooked of tangential direction.
Further, use the chromatic aberration coefficient k that is stored in the aberration coefficients table 38 R, k B, d Rx, d Ry, d Bx, d By, consider the chromatic aberration of pick-up lens 3 and pass through sample coordinate (x in (formula 3), (formula 4) calculating R plane, the B plane SR, y SR)(x SB, y SB).
[formula 3]
x sR y sR = k R O d Rx 0 k R d Ry x sG y sG 1 Formula 3
[formula 4]
x sB y sB = k B O d Bx 0 k B d By x sG y sG 1 Formula 4
At this, k R, k BBe the R when being benchmark, the multiplying power on B plane with Gr plane and Gb plane, d Rx, d RyBe the parallel offset amount on the R plane when being benchmark with Gr plane and Gb plane, d Bx, d ByIt is the parallel offset amount on the B plane when being benchmark with Gr plane and Gb plane.
Then, be of a size of 2560 * 1920 pixels, the location of pixels (u of corresponding color mosaic image if establish the color mosaic image s, v s) showing as under the xy coordinate system (formula 5), (formula 6), (formula 7).That is, when the color mosaic image is arranged formation by 2560 * 1920 equally spaced pixel, calculate
Figure GDA0000072699360000163
Give skew (u, v)=(1280,960) so that picture centre (u s, v s)=(1280,960) become initial point (x s, y s)=(0,0), obtain (formula 5), (formula 6), (formula 7).
[formula 5]
Figure GDA0000072699360000164
[formula 6]
[formula 7]
u sB v sB = x sB × 1600 + 1280 y sB × 1600 + 960 Formula 7
In addition, pixel is present in that (u, in the integer grid v), but the value of the sample coordinate of being calculated by (formula 5), (formula 6), (formula 7) is not limited to integer, is the real number that comprises fractional part.
Next, as shown in Figure 2, sampling portion 31 exports the value that is positioned at the sample coordinate pixel of being calculated by coordinate converting section 30 on every side respectively by the color mosaic image of buffer storage 21 storages by R pixel groups, Gr pixel groups, Gb pixel groups, B pixel groups.
In detail, be positioned at (u from the output of R pixel groups AR, V AR) on every side the value of pixel, be positioned at (u from the output of Gr pixel groups SG, v SG) on every side the value of pixel, be positioned at (u from the output of Gb pixel groups SG, v SG) on every side the value of pixel, be positioned at (u from the output of B pixel groups SB, v SB) on every side the value of pixel.
And, as described above, sampling location (u s, v s) be not limited to rounded coordinate, so read in encirclement (u s, v s) 4 values that the value pixel is arranged.
As shown in Figure 2, each plane in R plane, Gr plane, Gb plane, the B plane all is in length and breadth lattice point shape to have 4 the value pixel is arranged, and therefore, 4 that surround sample coordinate 401,402,403,404 have the value pixel, if this sample coordinate u SRBe (100.8,101.4) that (u v) is (100,100), (100,102), (102,100), (102,102), reads the value of these 4 pixels from buffering memory 21 then to surround 4 pixels of this sample coordinate.
Next, interpolating portion 32 is shown in Fig. 2 (f), obtain across the relative ratio that the distance between the value pixel is arranged of sample coordinate (at this, the x direction is 0.4: 0.6, the y direction is 0.7: 0.3), use 4 pixel values that the value pixel is arranged, by the pixel value of the R in interpolation calculation sampling location (100.8,101.4).
For example, to the R plane, with 4 pixel value R (100,100), R (100,102), R (102 that the value pixel is arranged, 100), R (102,102) expression, with the pixel value of the R of sampling location (100.8,101.4) with R (100.8,101.4) expression, then by R (100.8,101.4)=0.6*0.3*R (100,100)+0.6*0.7*R (100,102)+0.4*0.3*R (102,100)+and 0.4*0.7*R (102,102), the pixel value (100.8 of calculating the sample coordinate 401 on the R plane, 101.4), as the sampled value R of R Sample(u SR, v SR).
And, calculate pixel value in the sample coordinate by interpolation too by Gr pixel groups, Gb pixel groups, B pixel groups, be respectively Gr Sample(u SG, v SG), Gb Sample(u SG, v SG), B Sample(u SB, v SB).
Next, color generating unit 33 is according to the sampled value R of each color that is obtained by interpolating portion 32 Sample(u SR, v SR), Gr Sample(u SG, v SG), Gb Sample(u SG, v SG), B Sample(u SB, v SB), generate each color of pixel information (RGB composition) R (ud, vd), G (ud, vd), B (ud, vd)
At this moment, merely make R (ud, vd)=R Sample(u SR, v SR), B (ud, vd)=B Sample(u SB, v SB), G (ud, vd)=(Gr Sample(u SG, v SG)+Gb Sample(u SG, v SG))/2.
In addition, color generating unit 33 is also carried out the false colour inhibition.An example as the false colour inhibition, in the color image sensor that the Baeyer that is made of imaging apparatus 5 is arranged, near the high frequency treatment Nyquist frequency is easy to take place red and blue false colour, therefore, these speckles can be detected by the difference of getting Gr and Gb, false colour can be suppressed.
In detail, at first, calculate radio-frequency component K=Gr Sample(u SG, v SG)-Gb Sample(u SG, v SG).
Next, establish SumRB=R Sample(u SR, v SR)+B Sample(u SB, v SB), DiffRB=R Sample(u SR, v SR)-B Sample(u SB, v SB), use arithmetic expression DiffRB '=sign (DiffRB) Max (0, abs (DiffRB)-abs (K)), do not stride across 0 ground from abs (DiffRB) and deduct abs (K), calculate DiffRB '.Sign makes symbol be+operator of 1/0/-1, and abs is an operator of asking absolute value.
Next, use arithmetic expression R (u d, v d)=(SumRB+DiffRB ')/2 G (u, d, v d)=(Gr Sample(u SG, v SG)+Gb Sample(u SG, v SG))/2, B (u d, v d)=(SumRB-DiffRB ')/2, generate each color of pixel composition in the coloured image once more.Thus, can suppress the color generation of the red and blue false colour that takes place at radio-frequency head.
Next, carry out tint ramp (gamma) correction, colourity enhancing, the such image rectification of edge enhancing by vision correction portion 34, then, compressed with the method for JPEG (Joint Photographic Experts Group) etc. by the data image signal of compression unit 35 with coloured image, the data image signal after will being compressed by recording portion 36 stores in the recording medium.
Next, according to Fig. 4, illustrate by color mosaic image (input picture) and shake dynamic(al) correction and the order when distorting the anamorphose of aberration etc. and generating coloured image (output image) via image unit 2 input.This order is given command signal according to the program among the ROM19 of being stored in to each function portion by CPU18 and is carried out.In addition, the S among Fig. 4 represents step.In addition, anti-mosaic step of the present invention realizes its function by S180～S210, and anamorphose of the present invention realizes its function by S180.
At first, this order begins when by the operator image processing apparatus 100 having been imported enabling signal.
Then, in S110, obtain the color mosaic image, move to S120 then via image unit 2.
Then, in S120, the color mosaic image is read in buffer storage 21, corresponding with the Baeyer arrangement, store respectively by the picture element signal of R, the picture element signal of Gr, the picture element signal of Gb, the picture element signal of B, then, move to S190.In addition, planes of color decomposition step among the present invention and planes of color storing step are realized its function by means of S120.
On the other hand, in S130, use output pixel scanner section 29, scanning output image (coloured image) and order obtains process object pixel (u d, v d), then, move to S180.
In addition, in S140, use the corresponding camera lens state of camera lens state-detection portion's 37 detections and focal length and subject distance, then, in S150, obtain the aberration coefficients of preserving accordingly and zoom ratio etc. with the camera lens state from aberration coefficients table 38, then, move to S180.
In addition, in S160, use angular-rate sensor 15 and rock the rolling momentum that test section 40 detects camera head 1A, then, move to S170, in S170, obtain and be used to shake the parameter that momentum proofreaies and correct, then, move to S180.
Then, in S180, use coordinate converting section 30, the aberration coefficients that utilization obtains in S150, in S170, obtain rock correction parameter etc., to the location of pixels in the process object of the output image (coloured image) in S130, obtained, calculate and applied the coordinate (u that the anamorphose of distorting aberration and shake, zoom ratio etc. is handled s, v s), then, move to S190.In addition, the Coordinate Conversion step among the present invention realizes its function by means of S180.
Then, in S190, use sampling portion 31, from buffering memory 21 obtain be positioned at around the sample coordinate of calculating by coordinate converting section 30, each R, Gr, Gb, the pixel value of B then, moves to S200.
Then, in S200, use interpolating portion 32, by each R, Gr, Gb, B generates the pixel value R that is positioned at sample coordinate by the pixel value interpolation that is positioned at sample coordinate pixel on every side Sample(u SR, v SR), Gr Sample(u SG, v SG), Gb Sample(u SG, v SG), B Sample(u SB, v SB), then, move to S210.In addition, the interpolation procedure among the present invention realizes its function by means of S200.Then, in S210, use color generating unit 33 to synthesize the sampled value of each color of calculating by sampling portion 31, thus, handle the colouring information R (u that object pixel generates a plurality of colors by each d, v d), G (u d, v d), B (u d, v d), then, move to S220.In addition, the color among the present invention generates step and realizes its function by means of S210.
Then, in S220, judge in the output image to have or not next scanning element do not have pixel and move to S230 when (No) being judged as, when being judged as pixel (Yes), repeat S180～S220, in S220, become and move to S230 when not having pixel (No).
Then, in S230, use vision correction portion 34, the coloured image that is generated by color generating unit 33 is carried out tint ramp (gamma) correction, colourity enhancing, the such image rectification of edge enhancing, then, move to S240.
Then, in S240, use compression unit 35, to compress with JPEG methods such as (Joint Photographic Experts Group) via the data image signal of the coloured image of vision correction portion 34 output, the size of the view data when dwindling record then, moves to S250.
Then, in S250, service recorder portion 36 is stored in the data image signal after the compression in the recording medium such as flash memory, then, finishes this image processing program.
As mentioned above, the image processing apparatus 100 of first execution mode record and image processing method are by carrying out the buffering of view data required when carrying out anamorphose with the color mosaic image, the capacity of buffer storage 21 can be suppressed at 1 variable of per 1 pixel, can save the memory capacity and the frequency band of buffer storage 21, realize having the camera head 1 of colored mosaic and anamorphose with low-cost and low-power consumption.
(second execution mode)
Next, use Fig. 5～Fig. 8 that second execution mode of the present invention is described.Fig. 5 is the block diagram of structure of the camera head 1A of expression second execution mode of having used image processing apparatus of the present invention and image processing method, Fig. 6 is the flow chart of the order that generates of expression image processing apparatus of this second execution mode and the coloured image in the image processing method, and Fig. 7 is the flow chart of details of the order of compression in the presentation graphs 6 and speed buffering controller.
Camera head 1A in second execution mode has the structure identical with the camera head 1 shown in first execution mode basically, so to its identical same label of structure division mark and omit detailed explanation, below explanation becomes the part of feature.
Image processing apparatus 100A among the camera head 1A comprises: compression is from the compression unit 20 of the anti-mosaic image of image unit 2 outputs; Read in and be compressed and be kept at the extension section 25 that the anti-mosaic image in the buffer storage 21 stretches; From cache memory 27 or extension section 25 pixel data is outputed to the speed buffering controller 26 of sampling portion 31 according to the request that comes reflexive mosaic portion 28; And the cache memory 27 of the up-to-date pixel groups that obtains by speed buffering controller 26 of storage etc.
At first, with the color mosaic image that photographs from imaging apparatus 5 with raster order as being output to A/D converter 9 with the corresponding analog signal of the exposure of each pixel.
Next, A/D converter 9 is the digital signal (hereinafter referred to as pixel value) of 12bit with this analog signal conversion, and this pixel value is outputed to compression unit 20.
Next, compression unit 20 will be accumulated with the block unit of 8 pixels of level by each colored pixels from the pixel value group of A/D converter 9 order inputs, and the blocks of data that will be accumulated is compressed to 64bit from 12 * 8=96bit, and outputs to buffer storage 21.In addition, the planes of color decomposition portion among the present invention realizes its function by means of compression unit 20.In addition, the function of the compression unit among the present invention realizes that by means of compression unit 20 function of the extension section among the present invention realizes by means of extension section 25.
On the other hand, anti-mosaic portion 28 generates coloured image with raster order.At this moment, because carry out anamorphose, so via the sampling of the non-raster order of speed buffering controller 26 pairs of color mosaic images request.
Next, speed buffering controller 26 is accepted this sampling request, if the pixel value of the pixel of request has been kept in the cache memory 27, and accesses buffer 21 and read in this pixel value not then from cache memory 27, and output to sampling portion 31.
In addition, if the pixel value of the pixel of request is not kept in the cache memory 27, then 21 visits of 26 pairs of buffer storage of speed buffering controller contain the blocks of data of this pixel.
Next, buffer storage 21 will output to extension section 25 by the blocks of data of speed buffering controller 26 visits.
Next, extension section 25 stretches from the blocks of data of buffering memory 21 inputs, reproduces the pixel value of 8 12bit.
Next, speed buffering controller 26 receives the pixel value of 8 reproduced 12bit and outputs to the sampling portion 31 of anti-mosaic portion 28 from extension section 25, simultaneously, the pixel value of this 8 12bit is saved in the cache memory 27.In addition, if the space of cache memory 27 is full, then known speed buffering control method such as speed buffering controller 26 usefulness LRU (Least-Recently-Used) methods is upgraded.
Cache memory 27 and speed buffering controller 26 and anti-mosaic portion 28 are integrated as SRAM on same semiconductor integrated circuit, as compare with buffer storage 21 low capacity, at a high speed memory and install.In addition, the cache memory 27 of present embodiment is as reading the high-speed buffer action.In addition, the capable size of cache memory 27 constitutes the pixel value group that can preserve 1 piece.
Next, anti-mosaic portion 28 is identical with first execution mode, use is via each pixel data of the color mosaic image of speed buffering controller 26 input, in sampling portion 31, obtains each R, the Gr, the Gb that are positioned at around the sample coordinate of being calculated by coordinate converting section 30, the pixel value of B, then, in interpolating portion 32, by each R, Gr, Gb, B generates the pixel value R that is positioned at sample coordinate by the pixel value interpolation that is positioned at sample coordinate pixel on every side Sample(u SR, v SR), Gr Sample(u SG, v SG), Gb Ample(u SG, v SG), B Sample(u SB, v SB), then, in color generating unit 33, handle the colouring information R (u that object pixel generates a plurality of colors by each d, v d), G (u d, v d), B (u d, v d).
Next, according to Fig. 6, Fig. 7, illustrate by color mosaic image (input picture) and shake dynamic(al) correction and the order when distorting the anamorphose of aberration etc. and generating coloured image (output image) via image unit 2 input.This order is given command signal according to the program among the ROM19 of being kept to each function portion by CPU18 and is carried out.In addition, in this flow chart, give same numbering, and omit its detailed description the step identical with the flow chart shown in first mode.Compression step among the present invention is realized its function by S300, and stretching step of the present invention realizes its function by S500.
At first, this order begins when by the operator image processing apparatus 100A having been imported enabling signal.
Then, in S110, obtain the color mosaic image, move to the compression step of S300 then via image unit 2.
Then, in S300, shown in Fig. 7 (a), at first, in the piece segmentation procedure of S310, color mosaic image data are cut apart by each piece, then, moved to S320.At this moment, pressing R, the Gr of Baeyer arrangement, each pixel of Gb, B, is block unit with 8 pixels of vertical 1 pixel * level, and (x (n) n=1...8) as the pixel value group, carries out the processing of ensuing S320～S350 with above-mentioned 8 12bit pixel values.In addition, the planes of color decomposition step among the present invention realizes its function by S310.
Then, in S320, obtain minimum M in and the scope R (=maximum-minimum value) of the x (n) in the pixel value group, then, move to S330.
Then, in S330, use the scope R that in S320, obtains, use arithmetic expression E=log 2Shared index E is calculated in (R * 128/127)-6, then, moves to S340.At this moment, if E is a non-integer then with the E carry, if E is a negative, then making E is 0 and as nonnegative integer.
Then, in S340, use arithmetic expression M (n)=(x (n)-Min)/2 ECalculate the M of mantissa (n).At this moment,, then the M of mantissa (n) is rounding to integer, then, moves to S350 if the M of mantissa (n) is a non-integer.
Thus, the integer value of the desirable 12bit of Min, E desirable 0～7 is so available 3bit represents that the integer value of M (n) desirable 0～63 is so available 6bit represents.In addition, in the arithmetic expression of asking E of S330, owing to overflowing that rounding up in the step cause calculated by the multiply each other mantissa that relates to S340 of R and (128/127).Arrange these data and then be 63bit (12+3+6 * 8=63).
Then, in S350, compile these data 63bit (12+3+6 * 8=63) and with 8 pixel groups as packed data, then, move to the S400 of Fig. 6.Realize gray scale compression of the present invention by S310～S350.
Then, in S400, will in S350, compressed data be kept in the buffer storage 21, then, move to S500.Using under the situation of general DRAM as buffer storage 21, its great majority are with 8 * 2 nBit is as the visit unit, so append 1bit, it is efficient preserving with 64bit border (boundary).
Then, in S500, use extension section 25, this packed data is stretched as 8 12bit pixel value x ' (n), then, move to S600 from cushioning the packed data that memory 21 reads in 63bit.At this moment, in S500, read in data after the encapsulation (encapsulate) of 63 (or 64) bit that comprises desirable pixel data from buffering memory 21,12bit is decomposed into Min, 3bit is decomposed into E, is M (n) with 8 6bit data decomposition, by these decomposed data, make x ' (n)=M (n) * 2 E+ Min reproduces the pixel value group.
In addition, in the encapsulation step of S350, also can have the function of reversible compression unit.In compression unit 20, the data of 63bit are attempted reversible compression, if can not be compressed into below the 63bit, then reversible compression usage flag 1bit is arranged to false and gives, be kept in the buffer storage 21 as the data of 64bit.
In addition, under the situation that can be compressed into below the 63bit, reversible compression usage flag 1bit is arranged to true gives, then, compressed data are continued to be kept in the buffer storage 21.Thus, the rewriting amount of buffer storage 21 can be cut down, power consumption can be further realized saving.In addition, can use haul distance compression and entropy coding as reversible compression.
On the other hand, in the stretching step of S500,, at first confirm reversible compression usage flag from cushioning the packed data that memory 21 reads in 64bit, if reversible compression usage flag is false, then 8 pixel value x ' that represented with 12bit by ensuing 63bit output (n).In addition, if reversible compression usage flag is true, then 8 pixel value x ' that output after the reversible stretching of ensuing data is represented with 12bit (n).
Then, the speed buffering controller step of S600 is shown in Fig. 7 (b), according to the access request of coming reflexive mosaic portion 28, in S610, whether judgement is present in the cache memory 27 by the pixel data of anti-mosaic portion 28 requests, move to S620 when there be (yes) in pixel data being judged as, move to S611 when there be not (No) in pixel data being judged as.
Then, in S611, read in and comprise the packed data of asking pixel, then, move to S612.
Then, in S612, use extension section 25, Compression and Expansion data and obtain the pixel value group that comprises the pixel value of being asked then, in S613, are kept at the pixel value group that is stretched in the cache memory 27, simultaneously, move to S630.
On the other hand, when there be (yes) in pixel data in S610, in S620, read this pixel value, then, move to S630 from cache memory 27.
Then, in S630, will output to anti-mosaic portion 28, then, move to the S190 of Fig. 6 by the pixel value of anti-mosaic portion 28 requests.
Then, identical with first execution mode, in S180～S210, carry out anamorphose and anti-mosaic processing, then, after the compression step that has carried out the vision correction step of S230, S240, finish this processing.
As described above, the image processing apparatus 100A of second execution mode record and image processing method compress the color mosaic image data of preserving in the buffer storage 21 by using compression unit 20, can cut down the memory capacity of required buffer storage 21, and, be not that the color mosaic image of 8 multiple also can utilize general 8 * 2 efficiently by suitably setting compression ratio to the bit degree of depth nThe DRAM of width.
In addition, in anti-mosaic portion 28,, and under the situation of the adjacent pixels that generates coloured image, many for 1 pixel that generates coloured image with reference to the same pixel in (visit) color mosaic images with reference to the pixel value of a plurality of pixels in the color mosaic image.Corresponding therewith, the image processing apparatus 100A and the image processing method of the record of second execution mode have been introduced cache memory 27, so can cut down the frequency band of reading of buffer storage 21, can further cut down power consumption.
(variation)
More than, one embodiment of the present of invention have been described, but the present invention is not limited to the foregoing description, can get variety of way.
For example, in order more effectively to utilize cache memory, also can constitute the scanning sequency in the output pixel scanner section 29.That is, when whole pixel of scanning colour image, carry out regional strong scanning.As the example of such scanning, can list applying flexibly of hibert curve.For example, can be as shown in Figure 8, will export color images and be 8 * 8 piece, in each piece, scan along hibert curve, the coordinate of each pixel is outputed to coordinate converting section 30.In addition, if 1 interior picture element scan of piece finishes, then scan next piece by raster order.
In addition, compression unit 20 utilizes the compress mode based on the piece of vertical 8 pixels by as Motion JPEG or MPEG, can not need in the grating input required grating piece change.
In addition, in the camera head 1 of present embodiment, 1A, also comprise and make the test section of pick-up lens 3 in optical axis direction slide mechanism that slides and the position of detecting pick-up lens 3, in coordinate converting section 30, can carry out coordinate transform accordingly with the camera lens state of pick-up lens 3, be kept at the aberration coefficients in the aberration coefficients table 38 of image unit 2 with correction.

Claims (4)

1. image processing apparatus, its color mosaic image that has a monochromatic monochrome information by each pixel generates the monochrome information of a plurality of colors in each pixel, and implement the predetermined picture distortion and generate coloured image, wherein, above-mentioned color mosaic image is to be obtained by the imaging apparatus of one-board that different a plurality of color of light are carried out the pixel of light-to-current inversion that has in the image unit, it is characterized in that
Above-mentioned image processing apparatus comprises:
Planes of color decomposition portion, it resolves into above-mentioned color mosaic image a plurality of planes of color of the monochrome information that only comprises same color of light;
Buffer storage, the color mosaic image after its each planes of color storage according to above-mentioned a plurality of planes of color is separated by above-mentioned planes of color decomposition part;
The camera lens state of the pick-up lens in the aberration coefficients table, itself and above-mentioned image unit records the distortion aberration coefficients of the distortion aberration that is used to proofread and correct above-mentioned imaging apparatus accordingly;
Rock test section, it rocks the corrected value that rock of calculated signals with respect to the color mosaic image of above-mentioned image unit input according to the output of angular-rate sensor from above-mentioned image unit; And
Anti-mosaic portion, it generates having implemented the above-mentioned coloured image after the predetermined picture distortion from the color mosaic image of above-mentioned buffer storage output,
Above-mentioned anti-mosaic portion comprises output pixel scanner section, Coordinate Conversion portion, sampling portion, interpolating portion and color generating unit,
The such anamorphose of convergent-divergent of unified various aberration corrections of above-mentioned output pixel scanner section and jitter correction, image, and the pixel coordinate of the coloured image of output scanned;
Above-mentioned Coordinate Conversion portion rocks the signal that test section and above-mentioned aberration coefficients table are exported according to expression from the signal of the pixel coordinate of above-mentioned output pixel scanner section output with from above-mentioned, calculate the sample coordinate on the above-mentioned color mosaic image corresponding with the location of pixels of the above-mentioned coloured image of exporting
Above-mentioned interpolating portion is according to each planes of color, and according to the pixel value that obtains by above-mentioned sampling portion, interpolation generates each color pixel value under the sample coordinate,
Above-mentioned color generating unit is synthesized each color pixel value under the sample coordinate that is obtained by above-mentioned interpolating portion, generates the color data that each pixel has a plurality of color components.
2. image processing apparatus according to claim 1 is characterized in that,
Above-mentioned image processing apparatus also comprises:
Compression unit, its compression is from the above-mentioned color mosaic image of above-mentioned imaging apparatus; And
Extension section, it stretches by the above-mentioned color mosaic image that is stored in after the above-mentioned compressing section compresses in the above-mentioned buffer storage,
Above-mentioned compression unit is divided into a plurality of according to each planes of color of above-mentioned a plurality of planes of color, is compressed into below the pre-sizing according to this each piece that is split to form,
The color mosaic image of each piece of above-mentioned buffer memory stores after by above-mentioned compressing section compresses,
Above-mentioned extension section is according to each piece that is stored in the above-mentioned buffer storage, the color mosaic image after the Compression and Expansion,
Above-mentioned anti-mosaic portion generates above-mentioned coloured image according to the color mosaic image after being stretched by above-mentioned extension section.
3. image processing method, the color mosaic image that has a monochromatic monochrome information by each pixel generates the monochrome information of a plurality of colors in each pixel, and implement the predetermined picture distortion and generate coloured image, wherein, above-mentioned color mosaic image is to be obtained by the imaging apparatus of one-board that different a plurality of color of light are carried out the pixel of light-to-current inversion that has in the image unit, it is characterized in that
Above-mentioned image processing method comprises:
The planes of color decomposition step resolves into above-mentioned color mosaic image a plurality of planes of color of the monochrome information that only comprises same color of light;
The planes of color storing step is stored in color mosaic image after decomposing in the above-mentioned planes of color decomposition step according to each planes of color of above-mentioned a plurality of planes of color in buffer storage;
And
Anti-mosaic step generates having implemented the above-mentioned coloured image after the predetermined picture distortion from the color mosaic image of above-mentioned buffer storage output,
Above-mentioned anti-mosaic step comprises that output pixel scanning step, Coordinate Conversion step, sampling step, interpolation procedure and color generate step,
In above-mentioned output pixel scanning step, the such anamorphose of convergent-divergent of unified various aberration corrections and jitter correction, image, and the pixel coordinate of the coloured image of output scanned;
In above-mentioned Coordinate Conversion step, according to the signal that is illustrated in the pixel coordinate of exporting in the above-mentioned output pixel scanning step with from rocking the signal of test section and aberration coefficients table output, calculate the sample coordinate on the above-mentioned color mosaic image corresponding with the location of pixels of the above-mentioned coloured image of exporting
Wherein, the above-mentioned test section that rocks rocks the corrected value that rock of calculated signals with respect to the color mosaic image of above-mentioned image unit input according to the output of angular-rate sensor from above-mentioned image unit; The camera lens state of the pick-up lens in above-mentioned aberration coefficients table and the above-mentioned image unit records the distortion aberration coefficients of the distortion aberration that is used to proofread and correct above-mentioned imaging apparatus accordingly;
In above-mentioned interpolation procedure, according to each planes of color, according to the pixel value that obtains in above-mentioned sampling step, interpolation generates each color pixel value under the sample coordinate,
Generate in the step in above-mentioned color, each color pixel value under the sample coordinate that obtains in above-mentioned interpolation procedure is synthesized, generate the color data that each pixel has a plurality of color components.
4. image processing method according to claim 3 is characterized in that,
Above-mentioned image processing method also comprises:
Compression step, compression is from the above-mentioned color mosaic image of above-mentioned imaging apparatus; And
Stretching step is stretching in the above-mentioned color mosaic image that is stored in after the compression in the above-mentioned compression step in the above-mentioned buffer storage,
In above-mentioned compression step, be divided into a plurality of according to each planes of color of above-mentioned a plurality of planes of color, be compressed into below the pre-sizing according to this each piece that is split to form,
In above-mentioned planes of color storing step, in above-mentioned buffer storage, be stored in the color mosaic image of each piece after the compression in the above-mentioned compression step,
In above-mentioned stretching step, according to each piece that is stored in the above-mentioned buffer storage, the color mosaic image after the Compression and Expansion,
In above-mentioned anti-mosaic step, generate above-mentioned coloured image according to the color mosaic image after in above-mentioned stretching step, stretching.
CN2008101823405A 2007-11-28 2008-11-21 Image processing device and image processing method Expired - Fee Related CN101448169B (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2007307034 2007-11-28
JP2007307034A JP4504412B2 (en) 2007-11-28 2007-11-28 Image processing apparatus and image processing method
JP2007-307034 2007-11-28

Publications (2)

Publication Number Publication Date
CN101448169A CN101448169A (en) 2009-06-03
CN101448169B true CN101448169B (en) 2011-10-05

Family

ID=40678355

Family Applications (1)

Application Number Title Priority Date Filing Date
CN2008101823405A Expired - Fee Related CN101448169B (en) 2007-11-28 2008-11-21 Image processing device and image processing method

Country Status (5)

Country Link
JP (1) JP4504412B2 (en)
KR (1) KR101009108B1 (en)
CN (1) CN101448169B (en)
TW (1) TW200924512A (en)
WO (1) WO2009069453A1 (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5569042B2 (en) 2010-03-02 2014-08-13 株式会社リコー Image processing apparatus, imaging apparatus, and image processing method
JP5976315B2 (en) * 2011-12-28 2016-08-23 株式会社ザクティ Imaging device
CN104296664A (en) * 2014-09-17 2015-01-21 宁波高新区晓圆科技有限公司 Method for improving detection precision in geometric dimension visual detection
CN105847770B (en) * 2015-01-15 2017-08-08 展讯通信（上海）有限公司 The caching method of view data in picture processing chip and the chip
US10397489B2 (en) * 2015-05-15 2019-08-27 Sony Corporation Light source control device, method of controlling light source, and image capture system
KR102581210B1 (en) 2019-01-10 2023-09-22 에스케이하이닉스 주식회사 Method for processing image signal, image signal processor, and image sensor chip
CN113793249B (en) * 2021-08-09 2023-12-08 深圳曦华科技有限公司 Method for converting Pentille image into RGB image and related equipment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5652621A (en) * 1996-02-23 1997-07-29 Eastman Kodak Company Adaptive color plane interpolation in single sensor color electronic camera
US6091862A (en) * 1996-11-26 2000-07-18 Minolta Co., Ltd. Pixel interpolation device and pixel interpolation method
CN1780405A (en) * 2005-05-26 2006-05-31 智多微电子（上海）有限公司 Conversion and module from primitive Bayer interpolation to full color
CN101031021A (en) * 2006-02-27 2007-09-05 索尼株式会社 Image capturing apparatus, imaging circuit, and image capturing method

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005217896A (en) * 2004-01-30 2005-08-11 Shoji Kawahito Image encoding device
JP4815807B2 (en) * 2004-05-31 2011-11-16 株式会社ニコン Image processing apparatus, image processing program, and electronic camera for detecting chromatic aberration of magnification from RAW data
JP4438635B2 (en) * 2005-01-28 2010-03-24 セイコーエプソン株式会社 Image processing method, digital camera, and image processing program

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5652621A (en) * 1996-02-23 1997-07-29 Eastman Kodak Company Adaptive color plane interpolation in single sensor color electronic camera
US6091862A (en) * 1996-11-26 2000-07-18 Minolta Co., Ltd. Pixel interpolation device and pixel interpolation method
CN1780405A (en) * 2005-05-26 2006-05-31 智多微电子（上海）有限公司 Conversion and module from primitive Bayer interpolation to full color
CN101031021A (en) * 2006-02-27 2007-09-05 索尼株式会社 Image capturing apparatus, imaging circuit, and image capturing method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JP特开2005-217896A 2005.08.11
JP特开2006-211334A 2006.08.10

Also Published As

Publication number Publication date
TWI353779B (en) 2011-12-01
JP4504412B2 (en) 2010-07-14
WO2009069453A1 (en) 2009-06-04
CN101448169A (en) 2009-06-03
KR20100083197A (en) 2010-07-21
KR101009108B1 (en) 2011-01-18
JP2009130898A (en) 2009-06-11
TW200924512A (en) 2009-06-01

Similar Documents

Publication Publication Date Title
CN101448169B (en) Image processing device and image processing method
US7483058B1 (en) Video imaging system including a digital image sensor and a digital signal processor
US8249370B2 (en) Image processing method, image recording method, image processing device and image file format
KR101395433B1 (en) Imaging device and imaging method
US7265784B1 (en) Image processor with noise reduction circuit
KR100347508B1 (en) Photo Image Pick-Up Device
US20090073277A1 (en) Image processing apparatus, image processing method and image pickup apparatus
EP2184915B1 (en) Image processing device, image processing method, and image processing program
JP3946492B2 (en) How to generate a low resolution image
US7280705B1 (en) Tone correction method using a blending mask
RU2009136949A (en) VIDEO CAMERA
JP5793716B2 (en) Imaging device
US7355156B2 (en) Solid-state image pickup device, image pickup unit and image processing method
US20090066804A1 (en) Data-processing circuit and video camera
JP6014349B2 (en) Imaging apparatus, control method, and program
US7212214B2 (en) Apparatuses and methods for interpolating missing colors
US7499597B2 (en) Image processing device
JPH1169226A (en) Electronic camera
JP2001203969A (en) Image pickup device and its operation control method
US7139443B2 (en) Signal processing method for producing interpolated signal values in a sampled signal
JP2005277618A (en) Photography taking apparatus and device and method for correcting shading
JP3652123B2 (en) Image sensor, image processing apparatus, image processing system, and storage medium
JP3551670B2 (en) Electronic still camera
KR100267198B1 (en) Color image input device
KR100254081B1 (en) Apparatus and method for processing image data

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20111005

Termination date: 20171121

CF01 Termination of patent right due to non-payment of annual fee